Generative UI & Widgets

Most tools return text. A small, special family returns interface: the agent can draw an image, a form, a table, or a chart directly into the chat transcript in the user's browser. This chapter follows one render call from a backend ITool, across the AG-UI stream, into a React widget registry, and back — plus the acting Dashboard agent that uses the same machinery to drive a live UI.

The idea: tools that render, not narrate

A normal tool call is simple. It runs on the server, it returns a string, and the model reads that string. Everything happens in one place.

A render tool is different, because the work has to happen in the user's browser — that is where the pixels are. So the tool call takes a detour mid-flight:

  AGENT            SERVER                    BROWSER
    │
    │ "render_image(url)"
    ├──────────────▶ │
    │                │ tool call is PARKED
    │                │ (held open, waiting)
    │                │
    │                ├─── draw this ────────────▶ │
    │                │                            │ validates the args,
    │                │                            │ draws the widget
    │                │ ◀─── "shown it" ───────────┤
    │                │
    │                │ tool call UNPARKS
    │ ◀──────────────┤ and returns that
    │                     short acknowledgement
    │
    │ "Here's the diagram you asked for…"

The payoff is in what crosses each boundary. The model sends a URL, a form spec, or a set of rows. It gets back one short sentence like "Displayed the image to the user." The rendered widget itself never touches the model. No HTML, no image bytes, no component markup ever enters the conversation — so the return leg of a render call costs essentially nothing, however large the thing on screen is. (The arguments going out are still generated by the model and still cost tokens, so a thousand-row table is expensive to ask for, just not to receive.)

The backend render tools

Four ITools make up the family, all in Infrastructure.AI/Tools/ and registered by keyed DI in DependencyInjection.Tools.cs:

Each is opt-in per skill via allowed-tools in a SKILL.md, exactly like any other tool. They all reach the browser through IClientToolBridge — a mid-run client round-trip. If no client is attached (for example a console run), the tool fails gracefully with a plain message rather than hanging.

Dedup: SingleRenderProxyTool

Every render tool does the same five things: gate on the single render operation, check a client is attached, serialize the arguments to JSON, validate them, then invoke the bridge. Rather than repeat that scaffolding four times, the shared base class SingleRenderProxyTool (itself a BlockingProxyTool) owns it. A concrete tool supplies only what differs — its Name, Description, the two user-facing failure messages, and an optional ValidateArguments override:

if (!string.Equals(operation, RenderOperation, StringComparison.OrdinalIgnoreCase))
    return Task.FromResult(ToolResult.Fail($"Unknown operation: {operation}..."));

if (!IsClientAttached)
    return Task.FromResult(ToolResult.Fail(NoClientMessage));

var argumentsJson = JsonSerializer.Serialize(parameters, SerializerOptions);

var validationError = ValidateArguments(parameters, argumentsJson);
if (validationError is not null)
    return Task.FromResult(ToolResult.Fail(validationError));

return InvokeClientAsync(argumentsJson, TimeoutMessage, cancellationToken);

RenderImageTool, for instance, overrides ValidateArguments to reject anything that isn't an absolute https URL — so a javascript: or data: URI never reaches the browser. That's the first of two validation passes; the client registry validates again at the render boundary (defense in depth). Note the factory uses ToolResult.Fail(...) / .Ok(...) — there is no ToolResult.Success(...) factory (Success is a bool property).

The frontend registry — one entry per widget

On the browser side (Presentation.WebUI), every widget the agent is allowed to summon is one entry in a single file: src/features/chat/widgets/registry.tsx. If a widget is not in that file, the agent cannot draw it — which is the safety model from the top of this chapter, made concrete.

Each entry answers three questions, and you must supply all three:

• render — how do I draw this? A React component, given the agent's arguments.
• validate — are these arguments safe and well-formed? This runs at the trust boundary, on data the agent produced, before anything is drawn.
• ack — what do I tell the agent afterwards? The short sentence sent back to unpark the tool call.

export interface WidgetDefinition {
  render: (args: Record<string, unknown>) => ReactNode;
  validate: (args: Record<string, unknown>) => ValidationResult;
  ack: string;
}

// A Map — never a plain object — so an agent-influenced widget name can never
// resolve to an inherited member like "constructor" and get invoked.
const WIDGET_REGISTRY = new Map<string, WidgetDefinition>([
  ['render_image', { render: (a) => <AgentImage args={a} />, validate: parseImageArgs, ack: 'Displayed the image to the user.' }],
  ['render_form',  { render: (a) => <AgentForm  args={a} />, validate: parseFormArgs,  ack: 'Displayed the form to the user; their answers will arrive as their next message.' }],
  ['render_table', { render: (a) => <AgentTable args={a} />, validate: parseTableArgs, ack: 'Displayed the table to the user.' }],
]);

Generic dispatch — useAgentStream

The stream hook (src/hooks/useAgentStream.ts) drives the raw @ag-ui/client run. As tool-call events arrive it accumulates the streamed name and argument deltas per toolCallId; on TOOL_CALL_END it completes the call. Crucially, the hook is generic over the registry — it never names a specific widget:

const widget = getWidget(name);
if (!widget) return `The client has no handler for widget "${name}".`;

let args; try { args = JSON.parse(argsJson || '{}'); }
catch { return `The ${name} arguments could not be parsed.`; }

const check = widget.validate(args);   // trust boundary — validate the agent's args
if (!check.ok) return check.reason;    // invalid ⇒ render nothing, tell the agent why

useChatStore.getState().addMessage({ role: 'assistant', content: '', widget: { type: name, args } });
return widget.ack;                      // the string the agent observes as the tool result

The result is then POSTed back with postToolResult(threadId, callId, result), which unblocks the parked server-side tool so the same open run resumes with the agent's follow-up narration. A result is posted for every callId — even one with no matching handler — so the awaiting server tool never hangs. Because dispatch is generic, adding a fourth WebUI widget touches only registry.tsx; the hook is unchanged.

Widget persistence across reload

A rendered widget is part of the conversation, so it must survive a page reload. Two subtleties make this correct rather than merely working:

• A widget is persisted after the client confirms it rendered — not when the tool call is dispatched. Persisting on dispatch would leave a phantom widget behind if the round-trip timed out and nothing was ever drawn.
• The empty carrier messages that hold a widget (their text content is '') are filtered out of the history window the agent sees on its next turn (GetHistoryForDispatch). They belong to the rendered transcript, not to the model's text context — including them would feed the model blank turns.

The acting Dashboard agent

Presentation.Dashboard embeds an agent that doesn't just show UI — it acts on the dashboard. The same blocking-proxy machinery lets it read the current view state, set the time range, navigate, and refresh, and draw in-chat charts. The key files:

On the backend there are three tools behind this, each with one job:

• DashboardControlTool — the read and act operations, sent to the browser via IClientToolBridge.
• ListMetricsTool — hands the agent the curated catalogue of metrics, so it can only pick one that actually exists rather than inventing a name.
• RenderChartTool — draws an existing chart component from one of those metrics and returns a short summary.

There is nothing privileged about any of them. All three are opt-in per skill, and all three pass through the same GovernedAIFunction governance chain as every other tool in the harness — see Chapter 06 · Tools & Keyed DI.

Adding a new widget

1. Write the backend tool: derive from SingleRenderProxyTool, give it a ToolName const, a Name/Description, the two failure messages, and a ValidateArguments override that rejects unsafe input.
2. Register it by keyed DI in Infrastructure.AI/DependencyInjection.Tools.cs, injecting IClientToolBridge.
3. Add one entry to WIDGET_REGISTRY in registry.tsx: a render component, a validate parser, and an ack string. The stream hook needs no changes.
4. Grant it to a skill via allowed-tools.

Where to go from here