Execution API — Calling the Harness over HTTP
The Execution API is how another system drives the harness over ordinary HTTP. It does three things: it runs an agent you hand it as a zip file (bundles), it runs a graph of steps you define (workflows), and it tells you which tools your credential is actually allowed to invoke (tools). Nothing gets deployed and nothing gets installed. This chapter is written for whoever has to call that API, and it assumes you have never seen this codebase. Most of it covers bundles, which is the deepest of the three; workflows and tool discovery have their own sections near the end.
Start here: what this actually does
Every other way of using the harness assumes the agent is already there. Someone put an
AGENT.md file in a folder, the host found it when it started up, and from then on
you can talk to it. That works when you own both sides.
The Bundle API is for when you don't. Your customer wrote the agent. Your CI job generates one per build. A tenant uploads their own. In all of those cases you can't redeploy the host every time the agent changes — so instead, the agent travels with the request.
You upload a zip containing an agent, the harness gives you a receipt, you use that receipt to start a run, and then you either poll for the answer or watch it stream in live.
Here is the whole flow. Four HTTP calls, and only the first two are mandatory:
YOUR SYSTEM THE HARNESS
│
│ 1. POST /api/bundles stage the zip on disk,
│ (your bundle.zip) ────────▶ check it for nasties
│ ◀──────────── { handle: "abc123" } ← your receipt
│
│ 2. POST /api/bundles/abc123/runs work out what this caller
│ { userMessages: [...] } ────────▶ is allowed to do, then
│ ◀──────────── { jobId: "job789" } queue the conversation
│ ← "accepted", NOT "done"
│
│ 3. GET .../runs/job789 ...agent is thinking...
│ (poll until it's finished) ────────▶
│ ◀──────────── { status, result } ← the answer
│
│ 4. DELETE /api/bundles/abc123 ────────▶ throw the zip away
│ (optional — it expires anyway)
Four words appear over and over in this chapter. Learn them now and the rest reads easily:
AGENT.md describing the agent, plus
any skill files it needs. It is just a folder, zipped.
"a3f9c1…". It means "the copy of your zip that I unpacked and am holding for
you." You pass it in the URL of every later call. One handle can start as many runs as you
like.
Is this the surface you want?
The harness exposes three HTTP-shaped front doors and they are not interchangeable. Pick by who authored the agent:
| Surface | Host | Use it when |
|---|---|---|
| Bundle API | Presentation.ExecutionApi |
The agent is authored outside the host and shipped in with the request. One-off runs, tenant-supplied agents, CI jobs that carry their own agent definition. |
| AgentHub | Presentation.AgentHub |
The agent is already installed in the host and you want an interactive, stateful conversation (SignalR + AG-UI streaming, chat history). See its README.md. |
| MCP server | Infrastructure.AI.MCPServer |
You want to hand the harness's tools to someone else's agent, not run an agent here. See Chapter 08. |
The subsystem is gated on AppConfig:AI:BundleExecution:Enabled (default
false) — while it is off, every endpoint answers 403. Separately,
the host is fail-closed on authentication: it refuses to boot unless you
configure an Entra scheme (both TenantId and ClientId)
or consciously set Auth:AllowAnonymous: true. Setting exactly one of
tenant/client is treated as a mistake, not as an implicit request to serve openly, and
also refuses to boot.
What each call sends and gets back
Here are the real payloads, so you know exactly what to expect before you write any code.
1 — Upload the bundle
A multipart/form-data POST, the same thing an HTML file-upload form sends. The form
field must be named file — any other name is rejected.
{
"handle": "9f2c1d4a8b6e47f0913c2d5e7a8b0c1d",
"expiresAt": "2026-07-24T15:42:11.000+00:00"
}
expiresAt is the earliest the handle can disappear, not a fixed deadline —
see Lifetimes below for why the clock keeps getting pushed back.
2 — Start a run
You send the messages the agent should respond to. userMessages is a list because
you can seed a multi-turn conversation in one go.
{
"userMessages": [ "Hi, I am Sam. What can you do?" ],
"maxTurns": 10,
"stream": false
}
Only userMessages is required (1–100 entries, none of them empty).
maxTurns defaults to 10 and must be between 1 and 100;
stream defaults to false and is explained under
Two ways to get the output. A fourth field,
conversationId, turns the run into a continuing session — see
Continuing a conversation.
Continuing a conversation across runs
By default a run is self-contained: it answers the messages you send and forgets them. For a one-off question that is exactly what you want. For anything that runs as a session — a voice channel, a chat widget, a long-lived assistant — it means the agent starts every turn with no memory, so the only way to be understood is to resend the entire transcript each time. That gets more expensive and slower with every turn, and it stops working altogether once the conversation exceeds the 100-message cap.
Send a conversationId instead and the harness keeps the transcript for you.
The agent is given the conversation's recent history before the first turn, and this run's
turns are saved for the next one — so each request carries only what is new.
{
"userMessages": [ "And what about the second clause?" ],
"conversationId": "voice-session-8f14e45f"
}
The id is yours to choose — a GUID is the obvious pick — and it is created the first time you use it. A few things worth knowing:
-
The conversation belongs to whoever created it. Naming a conversation
that belongs to another caller returns
404, exactly as an unknown handle does, so nobody can probe for other people's session ids. - Turns never interleave. If a second run names a conversation that is mid-turn, it waits its turn rather than talking over it — including when the two runs land on different servers.
-
maxTurnsand the 100-message cap bound one run, not the conversation. A session can run far longer than either. What bounds its total length is the conversation's lifetime token budget, which is cumulative across every run; when it is exhausted the run stops gracefully withbudgetExhausted: truerather than failing. -
Only recent history is replayed — the last
AppConfig:AI:Conversations:MaxHistoryMessagesmessages (50 by default), so the prompt cannot grow without limit even though the stored transcript does.
{
"jobId": "1a2b3c4d5e6f7081",
"statusUrl": "/api/bundles/9f2c1d…/runs/1a2b3c4d5e6f7081",
"streamUrl": null
}
statusUrl and streamUrl are relative — prepend your API base URL
before using them. streamUrl is null unless you asked for
stream: true.
202 means "accepted", not "finished"
This is the single most common misreading of this API. HTTP 200 OK would
mean "here is your answer." 202 Accepted means "I have written your request
down and I will get to it." The agent has not said anything yet. The
response body contains no answer and never will — you have to go and fetch it in step 3.
3 — Get the result
Call statusUrl repeatedly (say, every second or two) until status
stops being "Queued" or "Running". While it is still working you get:
{
"jobId": "1a2b3c4d5e6f7081",
"status": "Running",
"error": null,
"createdAt": "2026-07-24T15:12:03.000+00:00",
"startedAt": "2026-07-24T15:12:04.100+00:00",
"completedAt": null,
"result": null
}
And when it is done, the same shape gains a result:
{
"jobId": "1a2b3c4d5e6f7081",
"status": "Succeeded",
"error": null,
"createdAt": "2026-07-24T15:12:03.000+00:00",
"startedAt": "2026-07-24T15:12:04.100+00:00",
"completedAt": "2026-07-24T15:12:09.780+00:00",
"result": {
"conversationSucceeded": true,
"finalResponse": "Hello Sam — I can answer short questions concisely.",
"turnCount": 1,
"totalToolInvocations": 0,
"budgetExhausted": false,
"conversationError": null
}
}
finalResponse is the agent's answer — that string is usually the only thing your
application cares about. The rest is diagnostics. Note what is not here: the messages
you sent, and the permissions the run executed under. The endpoint returns a fixed, deliberately
narrow subset of the run's internal record, so polling can never be used to read back what went
into the run — only what came out.
4 — Clean up (optional)
DELETE /api/bundles/{handle} returns 204 No Content and no body. You
can skip it entirely — unused handles are swept automatically — but deleting promptly frees
disk on the host and is good manners.
One handle can start any number of runs — upload once, ask ten different questions. The two also expire on separate clocks. Most importantly, the permissions a run executes under are decided when the run starts, from the credential that starts it — not when the bundle was uploaded, and not from whoever uploaded it. That one design decision is what makes a leaked handle harmless, and it is explained in full below.
Quickstart
This uses the anonymous development mode. Read the security section before pointing it at
anything shared. The commands below need curl, zip,
jq, and a bash-compatible shell — on Windows, Git Bash or WSL.
mkdir -p my-bundle/skills/greeter
cat > my-bundle/AGENT.md <<'EOF'
---
id: quickstart-agent
name: Quickstart Agent
description: Minimal externally-authored agent used to prove the bundle API end to end.
version: 1.0.0
skills: ["greeter"]
---
You are a concise assistant. Answer in at most three sentences.
EOF
cat > my-bundle/skills/greeter/SKILL.md <<'EOF'
---
name: "greeter"
description: "Greets the user and answers short questions."
version: "1.0.0"
---
Greet the user by name when they give one, then answer their question directly.
EOF
cd my-bundle && zip -r ../bundle.zip . && cd ..
# The host ships no launch profile, so pick the address explicitly:
# ASPNETCORE_URLS=http://localhost:5000 dotnet run --project src/Content/Presentation/Presentation.ExecutionApi
API=http://localhost:5000
# Anonymous dev mode needs no token. Against a real Entra host, add
# -H "authorization: Bearer $TOKEN"
# to every call below, where $TOKEN targets api://{ClientId}.
# 1 — register. The multipart field name MUST be "file".
HANDLE=$(curl -s -X POST "$API/api/bundles" -F "file=@bundle.zip" | jq -r .handle)
# 2 — start a run. 202 Accepted; the conversation has not happened yet.
JOB=$(curl -s -X POST "$API/api/bundles/$HANDLE/runs" \
-H 'content-type: application/json' \
-d '{"userMessages":["Hi, I am Sam. What can you do?"],"maxTurns":2}' | jq -r .jobId)
# 3 — poll until terminal.
curl -s "$API/api/bundles/$HANDLE/runs/$JOB" | jq '{status, result}'
# 4 — clean up (optional; the TTL sweeper would do it anyway).
curl -s -X DELETE "$API/api/bundles/$HANDLE" -o /dev/null -w '%{http_code}\n'
With a real Entra scheme configured, every call above additionally carries
-H "Authorization: Bearer $TOKEN", where the token targets
this API's own audience — api://{ClientId}. The bundle API never
shares an audience with the agent hub or the MCP server, because it runs code the host did not
write.
What goes in the zip
A bundle is the same on-disk shape the harness already understands for an installed agent, so anything you learned in Chapter 05 applies unchanged:
AGENT.md ← REQUIRED, at the archive root
skills/
research/SKILL.md ← optional; nested skills, owned by this agent alone
summarise/SKILL.md
plugins/
my-plugin/plugin.json ← optional; parsed and carried, not yet wired into the run
plugin.json ← optional; a root-level manifest is also read
Two hard requirements, both enforced at registration:
AGENT.mdmust sit at the archive root. Not in a subfolder. A zip built from a directory containing your bundle folder will be rejected — zip the folder's contents, as the quickstart does.- It must resolve to a non-empty id. The parser takes
id:from the YAML front matter, falls back toname:, and falls back again to the folder name. A manifest with no front matter at all yields no id and is rejected.
Everything else is optional. A malformed SKILL.md is skipped with a warning rather
than failing the whole bundle, and duplicate skill ids keep the first occurrence. Skills declared
in a bundle are owned by that bundle's agent: they are resolved only for its own
runs and never enter the host's global skill pool.
The security model: the bundle asks, the host grants
This is the most important section on the page. If you only properly read one, read this one.
Start with the problem. You are about to run a file that somebody else wrote, on your server. Inside that file, the author gets to write whatever they want — including a line that says which tools the agent is allowed to use:
---
id: totally-innocent-agent
allowed-tools: ["file_system", "shell", "http_client"]
---
If the harness simply believed that line, the security model would be "attackers, please declare only the permissions you ought to have." Obviously that cannot work. So the harness does the only sane thing:
allowed-tools in an uploaded AGENT.md is the bundle
asking. What it actually gets is decided entirely on the host side, from
configuration the bundle author cannot see or influence. The final permission set is
the overlap of the two — asking for more than you were granted gets you
nothing extra, and the request can only ever narrow the grant, never widen it.
The host-side half of that overlap is called the capability envelope. It is a block of config on the server that maps who is calling to what their agent may do:
{
"Envelopes": {
"Default": {
"AllowedTools": [],
"AllowedMcpServers": [],
"AutonomyCeiling": "Restricted"
},
"BySubject": {
"11111111-2222-3333-4444-555555555555": {
"AllowedTools": [ "file_system", "calculation_engine", "docs_search" ],
"AllowedMcpServers": [ "internal-docs" ],
"AutonomyCeiling": "Autonomous"
}
},
"ByRole": {
"BundleRunner.ReadOnly": {
"AllowedTools": [ "file_system" ],
"AllowedMcpServers": [],
"AutonomyCeiling": "Autonomous"
}
}
}
}
Resolution precedence, in order:
- Exact subject match (
BySubject) wins outright. - Otherwise the caller's matching roles are combined to the least-privilege result — the intersection of tool and MCP allowlists and the minimum autonomy ceiling. Holding more roles can only ever narrow a grant, never widen it.
- Otherwise the
Defaultapplies — which grants nothing unless an operator widened it.
There is no code path on which "no envelope" means "no restriction". An unrecognised
AutonomyCeiling string degrades to the most restrictive tier with a warning rather
than opening up, and a numeric or comma-composite value is rejected outright for the same reason.
The two allowlists gate different things, and you almost always need both:
AllowedMcpServers controls which servers may be
contacted; AllowedTools controls which tools may be
invoked — including the tools those servers publish. Granting
internal-docs lets the run reach that server and put its tool schemas in front of
the model; actually calling docs_search still requires
docs_search in AllowedTools, which is why it appears in the example
above alongside the two local tools.
An envelope with a non-empty AllowedMcpServers and an empty
AllowedTools is the one combination that looks permissive and behaves
inert. The run contacts every granted server, pays the round trip, publishes the fetched
schemas to the model — and then denies every resulting call, because none of those tool
names is on the invocation allowlist. The host logs a warning naming the granted servers
when it resolves an envelope in that shape. It warns rather than refusing to start:
the configuration fails closed, and an empty tool list is legitimate for a
caller who is meant to have no tool access at all.
Worked through end to end, with the config shown above, a run looks like this:
The uploaded AGENT.md asks for: file_system, shell, http_client
The caller's token says: sub = 1111…5555
→ host looks up BySubject["1111…5555"]
→ that envelope grants: file_system, calculation_engine, docs_search
(+ the internal-docs MCP server)
overlap (ask ∩ grant) = file_system
shell → DENIED (asked for, not granted)
http_client → DENIED (asked for, not granted)
calculation_engine → granted, but the bundle never asked, so it is
simply never wired into the agent
docs_search → granted, and reachable via the granted MCP server,
but this bundle never asked for it either
The agent runs with exactly one tool: file_system.
The denials there are bypass-immune: no auto-approve mode, no configuration flag, and nothing the bundle can say will turn them back on. That is what makes it safe to run an agent whose author you do not trust.
Autonomous ceiling suspends tool use entirely — it does not queue approvals
Live mid-tool-call approval routing is deferred, so the governor currently treats "this
action requires human sign-off" as a fail-closed block. In practice that means a
Restricted or Supervised ceiling stops the bundle from using
tools at all, rather than gating each call for approval. A bundle that must actually do
work therefore runs with AutonomyCeiling: "Autonomous" and is confined by
AllowedTools / AllowedMcpServers plus the host's own risk and
capability gates. This matches how the harness's plugin and tier baselines behave today —
see Security · Autonomy &
Governance for the tier model and
Security · Tools & Permissions
for the gates that still apply inside an Autonomous ceiling.
Which claim identifies you: oid vs sub
A JWT carries several different ids for the same person, and this API deliberately uses two different ones for two different jobs.
The symptom, when you get it wrong, is very specific: your BySubject grant
looks correct in config but behaves as if it isn't there, and every run falls through
to Default. That is almost always because the config was keyed on the wrong claim.
Ownership of handles and runs keys on the first available of
oid → object-identifier → nameidentifier → sub
→ name. Envelope lookup (BySubject) keys on
nameidentifier or sub only. For most Entra tokens these differ:
oid is the directory object id, sub is a per-application
pairwise identifier. If your BySubject grants appear to be ignored, you have
almost certainly keyed them on the object id. Key them on the sub claim, or
use ByRole with an app role — role values are read from the standard role
claim and from the short roles/role claim names Entra emits.
Three more properties worth designing around:
- MCP is allowlist-only and by name. A bundle references host-registered MCP servers by name and can never define an endpoint, so its entire outbound MCP surface is the list you granted. That closes SSRF by construction.
- Ownership is enforced on every operation. Run, poll, stream, and delete all
check that the caller owns the handle. A foreign or unknown resource is reported identically
—
404for reads, a silent204for delete — so the API never confirms that someone else's handle exists. - The grant follows the invoker, not the handle. A stolen handle runs under the thief's envelope, which is the point: it cannot be used to borrow the registrant's privileges.
Anonymous mode is a development mode
With Auth:AllowAnonymous: true the host authenticates every request as one synthetic
principal and logs a prominent warning at startup for as long as it runs. The consequences are
specific, and both matter:
- Every caller shares one owner. Ownership checks still run, but they all pass — so there is no isolation between clients. Anyone can run, poll, and delete anyone's handle.
- Every run gets the
Defaultenvelope. The synthetic principal carries no subject or role claims, so neitherBySubjectnorByRolecan match. IfDefaultis unconfigured, anonymous runs are granted nothing — the door is open, the room is empty.
Endpoint reference
The full machine-readable contract — schemas, examples, every status code — is checked in as
assets/openapi/bundle-api.yaml
(OpenAPI 3.1). Generate a client from it. This host serves no runtime Swagger
endpoint, unlike the agent hub.
This page and the spec are written by hand, not generated from the controller, so they
can drift. If anything here disagrees with the running server, the server is right —
please file an issue. (Changing the contract rather than consuming it? The canonical
update list is Change the wire contract in the host's
README.md.)
| Endpoint | Success | Notes |
|---|---|---|
POST /api/bundles |
201 |
Field name file. Location points at /api/bundles/{handle}, which answers only DELETE — there is no GET-a-bundle endpoint. 10 req/min. |
POST /api/bundles/{handle}/runs |
202 |
Body: userMessages (1–100, none empty), maxTurns (1–100, default 10), stream (default false), conversationId (optional; continues a stored conversation). |
GET /api/bundles/{handle}/runs/{jobId} |
200 |
Returns a fixed subset of fields — never echoes the envelope or your seed messages. |
GET …/runs/{jobId}/stream |
200 |
text/event-stream. Only valid for a run started with stream: true that is still Queued; otherwise 409. |
DELETE /api/bundles/{handle} |
204 |
Idempotent and non-disclosing — always 204. |
| Tools — discovery, and (opt-in) direct invocation | ||
GET /api/tools |
200 |
Lists the tools your credential may invoke — the intersection of what the host registers and what your envelope's AllowedTools grants. An empty list is a valid answer, not an error. |
GET /api/tools/{name} |
200 |
Describes one granted tool. A tool that doesn't exist and one you aren't granted both answer 404 — indistinguishably, by design. |
POST /api/tools/{name}/invoke |
200 |
Runs one operation of one tool and returns its result. Body: operation, optional parameters, optional timeoutSeconds. Needs the Harness.Tools.Invoke role and the tool in your envelope. Off by default — 403 until an operator enables it. |
| Workflows — submit a DAG, then run it | ||
POST /api/workflows |
201 |
Body: name, steps, edges, optional configuration. Every id is minted server-side. Cycles and unreachable steps are rejected here, not at first run. |
POST /api/workflows/{id}/runs |
202 |
No body. Returns a jobId and a statusUrl. 409 if this workflow already has a live run. |
GET /api/workflows/{id}/runs/{jobId} |
200 |
A projection — never echoes the envelope or tenant the run holds. |
GET …/runs/{jobId}/stream |
200 |
text/event-stream. First frame is always a SNAPSHOT. 503 when the host or you are at the stream ceiling. |
DELETE /api/workflows/{id}/runs/{jobId} |
200 |
Cancels the run and withdraws any approval it was waiting on. stopped: false means signalled, not yet stopped. 409 if already terminal. |
| Evals — run the host's own test suites | ||
GET /api/evals/datasets |
200 |
The names you are allowed to run. Empty list on a host with no dataset roots configured — that is the answer, not an error. |
POST /api/evals/runs |
202 |
Body: datasets (names, never paths), optional repeats, parallelism, tagFilter, failRateThreshold. Returns a jobId and a statusUrl. 404 if a name is not one this host serves. |
GET /api/evals/runs/{jobId} |
200 |
Status, plus counts/verdict/duration/cost once finished. Not the per-case transcripts. |
DELETE /api/evals/runs/{jobId} |
200 |
Cancels a run that has not started. stopped: false means it was already executing and will finish — an eval in flight cannot be interrupted. 409 if already terminal. |
Finding out what you may actually invoke
The envelope section above explains that the host decides what you may use. It does not tell you
what the host decided. GET /api/tools answers that directly, and it is the
first call worth making before you write a workflow with a ToolUse step.
curl -sS -H "Authorization: Bearer $TOKEN" http://localhost:5000/api/tools
{
"tools": [
{
"name": "document_search",
"description": "Searches ingested documents.",
"operations": ["search"],
"riskTier": "Low",
"isReadOnly": true,
"isConcurrencySafe": true,
"isDirectlyInvocable": true
}
]
}
The name is the exact string a ToolUse step must supply, and
operations tells you which values that step's operation field will accept — an
invocation naming anything else is rejected by the tool itself.
The shipped default envelope grants nothing, so an operator who has not configured a
grant for your credential leaves you with {"tools": []} and a
200. That is the fail-closed default working as intended. Ask your operator
for an Envelopes entry rather than assuming the host is broken. Equally,
this listing is yours — it is not the host's inventory, and another credential
will legitimately see a different set.
Two absences you may notice and should not chase. Tools that drive an interactive client —
dashboard_control and the render_* family — cannot be constructed in
this host at all, because there is no client on the other end to render into; they are omitted
rather than advertised as callable. And /api/tools is not the same thing as
the agent hub's /api/mcp/tools: that one lists tools published by external MCP
servers, which additionally require their server to be granted.
Running a single tool directly
Everything else in this chapter hands the host your work — a bundle, a workflow, a suite — and lets an agent decide which tools it needs along the way. This endpoint is the exception: you name the tool and the operation, and the host runs it. That is genuinely useful when you want one answer and a whole workflow would be ceremony. It is also the most privileged thing this API can do, so it is deliberately the hardest to switch on.
Bundle execution and workflow submission ship enabled in this host, because
serving them is what it is for. Direct invocation ships disabled, and
the difference is the point: those surfaces run an agent that chooses its own tools,
whereas this one lets a caller point at host-side code and pull the trigger. An operator
has to set AppConfig:AI:DirectToolInvocation:Enabled deliberately. Until
then every call answers 403, including yours with the right role.
Three separate things must all be true before a tool runs:
- You are authenticated.
- You hold the
Harness.Tools.Invokerole. Listing tools does not need it — seeing what the host could do and making it do it are different grants, and an operator can hand out the first without the second. - Your envelope's
AllowedToolsnames the tool. This one is checked twice, by two independent mechanisms.
curl -sS -X POST http://localhost:5000/api/tools/file_system/invoke \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"operation": "read", "parameters": {"path": "notes/todo.md"}}'
{
"tool": "file_system",
"operation": "read",
"succeeded": true,
"output": "- ship T2",
"outputTruncated": false,
"durationMs": 12
}
The status code and succeeded mean different things
This trips people up, so it is worth being blunt about. The HTTP status describes the
invocation; succeeded describes the tool's verdict. A tool that ran
perfectly and decided the answer was no returns 200 with
"succeeded": false. Every 4xx and 5xx below means the tool never ran at all.
400— the request is malformed, or names an operation the tool does not declare. The error names the operations it does accept.401— no credential, or a credential carrying nothing usable as an identity. Some identity providers emitsubvalues with characters the harness cannot use as a permission subject; the remedy is the same either way, which is to present a different token.403— you lack the role, or the host has this surface off, or governance refused an invocation of a tool you genuinely are granted.404— no such tool, or not granted to you, or not offered on this surface. Deliberately indistinguishable, for the same reasonGET /api/tools/{name}is.413— your request body exceeded the host's cap (64 KiB by default), refused before it was read.429— you already have the maximum number of invocations executing at once (4 per caller). This endpoint is capped by concurrency rather than request rate, because an invocation holds a server thread for its whole duration and a per-minute cap would not stop calls that are still running when the window rolls. Retry when one of yours finishes.504— the tool outran its deadline (30 s by default) and was cancelled. That deadline covers the whole invocation, not just the tool: authorization and the data-classification check happen under it too.
The most confusing 403 is the third kind. If your envelope's
autonomyCeiling is Supervised or Restricted, tool
execution is suspended entirely — mid-run approval routing is not built yet, so anything
needing approval is refused rather than queued. Only Autonomous permits invocation
today. That is a property of the envelope, not of this endpoint, and it applies to bundle and
workflow runs identically.
Some tools are listed but not invocable here
Check isDirectlyInvocable in the catalog before you build against a tool. When it is
false it is not saying you lack permission — it is saying the tool's
result would be meaningless to you. The render_* family and
dashboard_control return drawing instructions for a browser attached to a live agent
run, and there is no browser on your end of an HTTP call. delegate_task is excluded
for a different reason: it starts whole agent turns, so one apparently-simple synchronous call
would expand into open-ended work that outlives your request. Those tools stay listed because a
ToolUse step in a workflow can still use them perfectly well.
What comes back has been through a filter
Output is scrubbed by the host's response sanitizers before it reaches you, and so is a failing
tool's error message — error text is the likeliest place a filesystem path or a connection
string leaks out. Long output is cut at the host's ceiling and outputTruncated is set
to true; check that field, because a prefix that does not announce
itself will parse as a complete answer. Output is never compressed: the harness's
compression exists to squeeze results into a model's context window and does it by summarising
and leaving pointers an agent can expand, and you have no way to follow those.
Workflows: running a DAG instead of a conversation
A bundle run is a conversation. A workflow is a directed graph of steps — model calls, tool calls, retrieval, conditional branches, and human approval gates — that the harness executes for you. Use a workflow when the shape of the work is known in advance and you want it enforced; use a bundle when you want an agent to decide.
Submission and execution are separate calls. Submitting stores a definition under your ownership and costs nothing; running it spends the host's model and tool credentials.
# 1. Submit — the id comes back from the server, you do not choose it
curl -sS -X POST http://localhost:5000/api/workflows \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{
"name": "draft-and-review",
"steps": [
{"name": "draft", "type": "LlmCall",
"configuration": {"type": "LlmCall", "prompt": "Draft a summary."}},
{"name": "review", "type": "LlmCall",
"configuration": {"type": "LlmCall", "prompt": "Check it for errors."}}
],
"edges": [{"from": "draft", "to": "review", "type": "ControlFlow"}]
}'
# 2. Run it — returns 202 and a job id
curl -sS -X POST http://localhost:5000/api/workflows/$WORKFLOW_ID/runs \
-H "Authorization: Bearer $TOKEN"
# 3. Poll, or stream
curl -sS -H "Authorization: Bearer $TOKEN" \
http://localhost:5000/api/workflows/$WORKFLOW_ID/runs/$JOB_ID
Blocked counts as live
Starting a second run while one is queued, running, or parked on a human gate
returns 409. This is not a throttle you should retry past — two runs of one
workflow would share a single execution state machine, so the second would re-execute
the first's live steps and adopt its outputs. Wait for the first to finish, or cancel it.
That makes status worth reading carefully. Treat only Succeeded,
Failed, and Cancelled as final. Blocked means the run is
parked on a human approval gate: it keeps its job id, keeps its hold on the workflow, and
resumes on that same id once someone answers the gate. A client that treats Blocked
as an ending will give up on a run that was about to continue.
Streaming a workflow run
GET …/runs/{jobId}/stream opens a text/event-stream. Unlike the bundle
stream, you may attach at any time — the first frame is always a SNAPSHOT of the
run as it stands, so there is no window in which you have to reconcile what you missed.
| Frame | Meaning |
|---|---|
SNAPSHOT | Always first. The run's state when you attached; isTerminal: true means it had already finished and nothing further follows. |
STEP | A step started or completed. sequence is monotonic within the run. |
FINISHED | Terminal. The stream closes after it. |
PARKED | The run hit a human gate. The stream closes but the run has not ended — re-attach after the gate is answered. |
GAP | You fell behind and droppedCount frames were discarded. Reported rather than hidden, so a slow client never mistakes a truncated stream for a complete one. |
One property to design around: authorization is checked when the stream opens, not continuously. A token revoked mid-run keeps receiving until the run ends. The exposure is bounded by the run's duration, and the frames carry nothing you could not already read from the status endpoint — but it is a deliberate trade, not an oversight.
Evals: running the host's own test suites
The other three surfaces run your work — your bundle, your workflow, your tool call. This one runs the host's. An evaluation suite is a file of test cases an operator has put on the server, and running it asks the harness to answer every case and score itself. You use it to check that a change to prompts, models, or tools did not make things worse.
It is off unless someone turned it on, and gated on a role. Every endpoint
below 403s until AppConfig:AI:Evaluation:Enabled is set, and the host
refuses to start if that is set without at least one dataset root. On top of that you need the
Harness.Evals.Execute role — this is the only surface in this chapter that
asks for one. The reason is that every other route runs your work on your
grant, whereas this one spends the host's model budget on the operator's suites. Holding a valid
token is not the same authority as being allowed to do that.
The role does not change ownership: it decides whether you may evaluate at all, and you still only ever see your own runs.
You name a dataset; you never give a path
This is the part worth understanding, because it looks like an inconvenience and is actually the
whole security design. The command underneath takes file paths — which is right for the
command-line runner, where a developer points it at a file on their own machine. Over HTTP that
same shape would mean every request carried a filesystem reference, and the only thing standing
between a caller and reading /etc/shadow would be a check remembering to say no.
So the wire contract has no path field at all. You send a name; the server finds the file by listing the directories an operator configured and matching what is actually there. A name cannot express “outside those directories”, so the dangerous request is not rejected — it cannot be written down. Ask for what is available first:
curl -sS -H "Authorization: Bearer $TOKEN" http://localhost:5000/api/evals/datasets
# { "datasets": ["router-accuracy", "safety-refusals"] }
Those names are the files sitting directly in a configured root, without their extensions. Subdirectories are not searched: a name with a slash in it would be a path again, just wearing a different hat.
Start a run, then poll it
JOB_ID=$(curl -sS -X POST http://localhost:5000/api/evals/runs \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{"datasets":["router-accuracy"],"repeats":3}' | jq -r .jobId)
curl -sS -H "Authorization: Bearer $TOKEN" \
http://localhost:5000/api/evals/runs/$JOB_ID
The 202 means the run was accepted, not that it finished — a suite takes as long as
it takes. Once it is done the poll carries a report: how many cases passed, failed,
warned, or errored, the overall verdict, how long it took, and what it cost in dollars.
The report is also filed into the host's own evaluation store automatically, so it shows up in the dashboard without you posting anything back — that is the main thing running a suite server-side buys you over running the CLI.
The poll does not carry the per-case transcripts. Those contain every test input and the agent's full response, which is a lot of data to put on a status check and more of the agent's output than any other endpoint here exposes. If you need transcripts, read the report files the evaluation framework writes on the server.
Succeeded does not mean everything passed. It means the
evaluation ran. A suite where every case failed is still a succeeded run — one
whose report says "verdict": "Fail". The distinction matters: if the two were
collapsed you could not tell “my prompt change broke things” from “the
host is broken and never answered me”.
Two limits to know about
Cost is capped before anything runs, not interrupted afterwards. The host
multiplies the number of cases by your repeats and refuses the whole run if that
exceeds MaxCaseExecutionsPerRun. It refuses rather than quietly running a subset,
because a pass rate for a suite that only half-ran is worse than no answer.
Cancelling only works before the run starts. DELETE on a queued
run stops it. On a run already executing you get 200 with
"stopped": false, and it finishes anyway. That is deliberate honesty rather than a
gap: a workflow can be signalled to stop mid-flight, an evaluation cannot, and telling you it
had stopped when the spend was still accruing would be the worse answer. The real protection is
the cost cap above.
Two ways to get the output
Background + poll (the default)
Your run joins a queue, and a background worker — the dispatcher — picks runs off that queue and executes them. You find out how it went by polling.
One property of the dispatcher matters a great deal when you set your poll interval: it drains the queue one run at a time. Your run waits behind every other queued run on that host. So the time to a result is not just how long the model takes to think — it is that plus however long the queue ahead of you takes. Budget for both.
Poll until status is Succeeded or Failed, then read
result. The distinction that catches people:
Succeeded describes the run, not the answer
A conversation that completed but whose agent reported a failed turn is
status: "Succeeded" with result.conversationSucceeded: false and
the reason in result.conversationError. Only an unhandled exception, a handle
that expired before the run started, or cancellation produces
status: "Failed" — and then the reason is in the top-level
error, always a stable scrubbed string, never a raw exception. A correct
client checks both fields.
Live stream (opt in with stream: true)
Instead of polling, you hold one long-lived HTTP connection open and the server pushes output
down it as the agent produces it. The format is Server-Sent Events (SSE) — a
plain-text streaming format with the content type text/event-stream, supported by
every browser and most HTTP clients.
A streaming run behaves differently from a background one in a way that surprises people:
it is reserved, not queued. Nothing runs when you POST it. The run sits
waiting, and opening streamUrl is what actually starts the agent. Closing
that connection cancels the run. If you never connect at all, the reservation is thrown away
after StreamReservationTtl (default 5 minutes).
Frames are data: {json}\n\n, camelCase, with a type discriminator taken
from the AG-UI wire vocabulary — a hand-written client that switches on type works
unchanged. A strict AG-UI client library is not guaranteed to: tool-call frames can arrive while
a TEXT_MESSAGE is still open (see below, some AG-UI client libraries reject that),
and TOOL_CALL_RESULT's fields have no AG-UI precedent to match against.
Frame type | Payload | When |
|---|---|---|
RUN_STARTED | threadId (the handle), runId (the job id) | Once, first. |
TEXT_MESSAGE_START | messageId, role: "assistant" | Lazily, on the first real token. A run that emits no text emits no message frames at all. |
TEXT_MESSAGE_CONTENT | messageId, delta | Per token batch. Append to your buffer. |
TEXT_MESSAGE_END | messageId | Once, if the message was opened. |
TOOL_CALL_START | toolCallId, toolCallName | When the agent decides to call a tool. May arrive before the first TEXT_MESSAGE_START. |
TOOL_CALL_ARGS | toolCallId, delta, withheld (only present when true) | Immediately after TOOL_CALL_START — the complete arguments JSON in one frame, not an incremental delta. |
TOOL_CALL_END | toolCallId | Immediately after TOOL_CALL_ARGS. |
TOOL_CALL_RESULT | toolCallId, result (redacted preview) | Once the tool has actually run. |
RUN_FINISHED | threadId, runId | Terminal, on success. |
RUN_ERROR | message (caller-safe) | Terminal, on failure. Never both this and RUN_FINISHED. |
Two things about the tool-call frames that are easy to assume wrong: the delta
field on TOOL_CALL_ARGS is named to match the AG-UI wire vocabulary, but it is
not incremental — the model-provider integration only ever exposes a tool
call's arguments once fully assembled, so all three of TOOL_CALL_START,
TOOL_CALL_ARGS, and TOOL_CALL_END arrive back-to-back in one burst.
And tool-call frames freely interleave with text frames, including arriving before the
assistant message ever opens — a client cannot assume text comes first.
TOOL_CALL_ARGS.delta and TOOL_CALL_RESULT.result are never the raw
payload — both pass through the same secret-redaction the harness applies before persisting
tool activity to its observability store, since a live stream to a browser is just as much
an exposure point as disk. They differ in how they bound size: result is
truncated to a fixed preview length, which is safe for free text. delta is
never truncated — cutting JSON mid-token would hand the client invalid data — so above a
size ceiling the real arguments are withheld instead: delta
becomes the fixed placeholder "{}" and withheld is true.
A client should treat a withheld frame as "arguments unavailable," not attempt
to interpret "{}" as the tool's real (empty) arguments. If a tool call fails,
result is a generic failure message, not the underlying exception text.
async function runStreamed(API: string, handle: string, token: string): Promise<string> {
const start = await fetch(`${API}/api/bundles/${handle}/runs`, {
method: 'POST',
headers: { 'content-type': 'application/json', authorization: `Bearer ${token}` },
body: JSON.stringify({ userMessages: ['Walk me through it.'], maxTurns: 5, stream: true }),
});
const { streamUrl } = await start.json();
// Opening this is what runs the agent. EventSource cannot send an Authorization
// header, so use fetch + a stream reader when the host requires a bearer token.
const res = await fetch(`${API}${streamUrl}`, { headers: { authorization: `Bearer ${token}` } });
if (!res.body) throw new Error(`Stream failed: ${res.status}`);
const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
let buffer = '', answer = '';
for (;;) {
const { value, done } = await reader.read();
if (done) break;
buffer += value;
// Frames are separated by a blank line.
let cut;
while ((cut = buffer.indexOf('\n\n')) !== -1) {
const frame = buffer.slice(0, cut).replace(/^data: /, '');
buffer = buffer.slice(cut + 2);
const evt = JSON.parse(frame);
if (evt.type === 'TEXT_MESSAGE_CONTENT') answer += evt.delta;
if (evt.type === 'TOOL_CALL_START') console.log(`calling ${evt.toolCallName}…`);
if (evt.type === 'RUN_ERROR') throw new Error(evt.message);
if (evt.type === 'RUN_FINISHED') return answer;
}
}
// The stream ended without a terminal frame — the connection dropped mid-run.
throw new Error('Stream ended before RUN_FINISHED or RUN_ERROR.');
}
Streams are capped by concurrency, not rate: each open connection holds a permit
for its whole lifetime, and a caller may hold MaxConcurrentStreamsPerCaller
(default 4) at once. The excess is rejected with 429 immediately rather than queued,
so a client that opens streams in a loop must bound itself.
What the host does with your archive
Nothing in the zip is parsed until every structural guard has passed, and any failure deletes the
partial extraction before returning. Each rejection is a 400 whose
detail names the guard — never the archive's contents.
| Guard | Default limit | Setting |
|---|---|---|
| Archive size on the wire | 10 MiB | MaxArchiveBytes (also caps the multipart body) |
| Entry count | 2 000 | MaxEntryCount |
| Total uncompressed size | 50 MiB | MaxTotalUncompressedBytes |
| Compression ratio | 100× | MaxCompressionRatio — only applied once a bundle expands past 1 MiB, so small, highly-compressible markdown bundles are never caught by it |
Zip-slip (../ or absolute entry paths) | rejected | — |
| Symlinks resolving outside the staging directory | rejected | — |
| Not a zip / empty archive | rejected | — |
The decompression-bomb check runs twice: once cheaply against the archive's declared header sizes, then again against actual bytes written during extraction — so an archive that lies about its entry lengths still trips it.
Accepted bundles are extracted into their own unique subdirectory of TempRoot.
There is one rule about where that root may live, and it matters:
TempRoot must not sit inside a skill or agent discovery pathThe host scans its discovery folders recursively to find skills. So if you stage uploaded bundles inside one of those folders, the host will happily walk into the staging area, find a customer's private skills, and publish them globally to every agent on the box. That is a serious leak, so the staging service checks for the overlap up front and simply refuses to stage rather than letting it happen.
Errors you should handle
| Status | Means | Do |
|---|---|---|
400 | Archive guard rejection, or an invalid run request (no messages, empty message, maxTurns out of range, malformed conversationId — letters, digits, hyphens and underscores only, up to 200 characters). | Read detail; fix the payload. Not retryable as-is. |
401 | No/invalid token, or a principal with no usable identity to own resources under. | Re-authenticate against api://{ClientId}. |
403 | Bundle execution is disabled on this host. | Operator action — Enabled: true. Do not retry. |
404 | Handle, run, or named conversation is unknown, expired, or not yours. These are deliberately indistinguishable, so the API never confirms that someone else's resource exists. | Re-register the bundle to get a fresh handle; check the conversationId is one you created. |
409 | The run is not awaiting a stream — it is a background run, already being streamed, or already finished. | Poll the status endpoint instead. |
429 | 10/min register; 60/min run, poll, and delete; the stream endpoint is capped by open connections instead. | Back off; for streams, close one first. |
500 | Unexpected server error. The body is deliberately generic. | The detail is in the host's logs. Retry idempotently. |
Lifetimes
| Clock | Default | Behaviour |
|---|---|---|
HandleTtl | 30 min | Sliding — refreshed on every lookup and every run start. expiresAt is a floor, not a deadline. |
RunRecordTtl | 30 min | How long a completed run stays pollable before it is swept. Read your result inside this window. |
StreamReservationTtl | 5 min | How long an unclaimed streaming reservation waits for you to connect. Independent of RunRecordTtl on purpose. |
CleanupInterval | 60 s | How often the sweeper deletes expired staging directories and run records. |
An in-flight run holds a lease on its staged bundle, so the sweeper can never delete the directory out from under a running agent.
Handles, run records, and results live in memory on one host. Two consequences follow, and both will bite you in production if you don't plan for them. Restart the host and every in-flight and completed run is gone. Put a load balancer in front of two instances and your poll will eventually land on a machine that has never heard of your job.
So until these stores are made durable, your client has to be the system of record. Concretely, that means two things: save the final response yourself the moment you read it, and keep one caller pinned to one instance — or just run a single instance.
Configuration reference
All of it lives under AppConfig:AI:BundleExecution.
| Key | Default | Purpose |
|---|---|---|
Enabled | false | Master toggle. Off ⇒ every endpoint 403s. |
TempRoot | "" (system temp) | Staging root. Must be disjoint from all skill/agent discovery paths. |
MaxArchiveBytes | 10485760 | Wire size cap; also bounds the multipart body. |
MaxEntryCount | 2000 | Entry-count cap. |
MaxTotalUncompressedBytes | 52428800 | Absolute expansion cap. |
MaxCompressionRatio | 100 | Expansion-factor cap (above a 1 MiB floor). |
HandleTtl | 00:30:00 | Sliding handle lifetime. |
RunRecordTtl | 00:30:00 | How long a terminal run stays pollable. |
StreamReservationTtl | 00:05:00 | How long an unclaimed stream reservation survives. |
MaxConcurrentStreamsPerCaller | 4 | Open streams one caller may hold. |
CleanupInterval | 00:01:00 | Sweeper period. |
Envelopes | fail-closed | The per-caller grant table (see above). |
Auth:TenantId / Auth:ClientId | unset | This API's own Entra audience. Supply both or neither. |
Auth:AllowAnonymous | false | Explicit local-development opt-in. Cannot be combined with a configured scheme. |
The eval endpoints read a separate section, AppConfig:AI:Evaluation.
| Key | Default | Purpose |
|---|---|---|
Enabled | false | Master toggle for /api/evals. Off ⇒ every endpoint 403s. |
DatasetRoots | [] | Directories datasets may be read from. Use absolute paths. Setting Enabled without any root refuses to start. |
MaxDatasetsPerRun | 10 | Datasets one run may name. |
MaxCaseExecutionsPerRun | 500 | Cases × repeats — what the run actually costs. Refuses, never truncates. |
MaxDatasetBytes | 5242880 | Checked before the file is opened, so an oversized dataset is never parsed. |
MaxRepeats | 50 | Ceiling on repeats. |
MaxParallelism | 128 | Ceiling on parallelism. |
Running the host
ASPNETCORE_URLS=http://localhost:5000 \
dotnet run --project src/Content/Presentation/Presentation.ExecutionApi
The shipped appsettings.json enables bundle execution but leaves Auth
empty, so a fresh clone boots only in Development, where the shipped
appsettings.Development.json opts into anonymous auth. Every other environment
trips the fail-closed guard above at startup until you configure Entra. Outside Development the
host also enables HSTS and HTTPS redirection.
Troubleshooting
| Symptom | Cause |
|---|---|
| Host throws at startup about authentication | Fail-closed guard. You configured neither a scheme nor AllowAnonymous, set exactly one of TenantId/ClientId, or set both a scheme and AllowAnonymous (contradictory). |
Every call returns 403 | Enabled is false. |
Register returns 400 "Bundle has no AGENT.md at its root." | You zipped the parent folder. Zip the bundle directory's contents. |
Register returns 400 about the staging root | TempRoot overlaps a skill or agent discovery path. |
| Runs succeed but the agent never uses a tool | The envelope. Either the tool is not in AllowedTools, or the ceiling is not Autonomous — see the autonomy callout. |
BySubject grant appears ignored | Keyed on oid instead of sub/name-identifier. |
Stream returns 409 | The run was not started with stream: true, or something already claimed it. |
Poll returns 404 for a job you just started | Different caller identity than the one that registered the handle, a restarted host, or the record was swept. |
Where to go from here
The tools an envelope can grant, and the governance chain every one of them passes through.
05 · RelatedThe SKILL.md format your bundle's nested skills use, unchanged.
Where running externally-authored code sits in the harness's overall security posture.