AI platform¶

The AI platform is every admin surface that governs how Datahub runs HERC, the agent system, and any LLM-powered feature — which models, with which keys, under what guardrails, with what observability, and what they cost.

It isn't one screen. It's a family of surfaces under /ai/*, and this page is the map.

ProvidersOpenAI · Anthropic · Azure OpenAI · Databricks · Private AI

Admin surfaces6 under /ai/*

Default guardrails6 rules

Keys at restFernet-encrypted

When to reach for this¶

Add or rotate a provider key

Store an OpenAI / Anthropic / Azure OpenAI / Databricks key per tenant, or switch on the platform-hosted private model.
Give an agent a cheaper or stronger model

Per-agent overrides for provider, model, temperature, and token ceilings — no code change.
Build an agent for your own data

Say what it is for, which tools and data it may use, its model, and who may call it. It answers once a reviewer approves it.
Block PII, secrets, and prompt injection

A content-safety pipeline that runs before and after the LLM, direction- and agent-scoped.
See every LLM call

Latency, tokens, prompt, response, guardrail violations, and evals — one trace per call.
A/B test before flipping the default

Run an agent against a fixed dataset to compare model versions and catch prompt regressions.

When the AI platform is not the surface

Genie chat on dashboards is proxied through the Databricks workspace integration — it needs workspace credentials, not an AI provider key.
Governing which MCP tools an agent may call lives in MCP governance, not here.
The knowledge an agent reasons over is built once per tenant in Organisation DNA.

Where each task lives¶

You want to…	Go to
Add or rotate an OpenAI / Anthropic / Azure OpenAI / Databricks API key	Provider keys
Switch on the platform-hosted private model (Granite-4)	Provider keys + license
Change the default model for an agent (e.g. cheaper / faster)	Model settings
Build an agent for your own data (tools, data scope, model, who may call it)	Agents
Block PII, secrets, prompt injection, off-topic prompts	Guardrails
See every LLM call (latency, tokens, prompt, response, evals)	Observability
Run an agent against a fixed dataset to compare versions	Experiments

How the AI platform works¶

Every LLM-backed feature — a HERC turn, a metric suggestion, a dashboard generation, a transcription — funnels through the same path: resolve the model for the calling agent, decrypt the provider key, run the prompt through input guardrails, call the provider, run the response through output guardrails, and record the whole thing as a trace.

flowchart LR
    H["HERC / agent turn"] --> AIP
    M["Module feature<br/>metric · term · dashboard · transcription"] --> AIP
    AIP(["AI platform"]) --> RES["Resolve model<br/>agent > global > default"]
    RES --> KEY["Provider key<br/>Fernet-decrypted"]
    KEY --> GIN["Input guardrails<br/>PII · secrets · injection"]
    GIN --> PROV["Provider<br/>OpenAI · Anthropic · Azure OpenAI · Databricks · Private AI"]
    PROV --> GOUT["Output guardrails<br/>redact · moderate · disclaimer"]
    GOUT --> R["Response"]
    AIP -.every span.-> OBS[("Observability<br/>Phoenix · OTel traces")]
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class AIP accent

Figure 1 — One request path for every LLM feature. Guardrails wrap the provider call on both sides, and every span is traced to observability regardless of which module started the call.

Why teams reach for this

You get one governed pipeline for every LLM in the product: rotate a key once and every agent picks it up, add a guardrail once and it protects HERC and metric suggestions and transcription, and read a single trace browser to answer "where did this prompt go and what did it cost?"

The surfaces¶

Surface	Route	What it governs
Provider keys	`/ai/provider-keys`	Per-tenant API keys, transparent fallback, Private AI license + token metering.
Model settings	`/ai/model-settings`	Global default and per-agent model + parameter overrides; the result limits for Genie and warehouse queries.
Agents	`/admin/ai/agents`	Your own custom agents: what each may use and read, who may call it, and which version is live after review.
Guardrails	`/ai/guardrails`	Input/output content-safety rules, direction- and agent-scoped.
Observability	`/ai/observability`	Trace browser, evals, and annotations backed by Arize Phoenix.
Experiments	`/ai/experiments`	Offline evaluation of agents against fixed datasets.

Provider keys — `/ai/provider-keys`¶

Datahub talks to LLM providers using per-tenant API keys stored encrypted in the database (Fernet, derived from the platform secret). One tenant can hold multiple active keys for the same provider; the most recent is preferred. Encryption is resilient — a corrupted key is skipped (with a warning), so one bad row never breaks the rest.

Supported providers:

Provider	Use cases	Notes
OpenAI	Default for HERC, agents, transcription (Whisper).	Standard `OPENAI_API_KEY`.
Anthropic	Alternative LLM (Claude Sonnet / Haiku). Useful as a fallback.	The platform automatically chooses sensible defaults per provider.
Azure OpenAI	HERC, agents, Insights, glossary enrichment, document processing and the semantic layer on your own Azure resource.	Endpoint, default deployment and an optional API version per key. Not yet the Brain or transcription: see Azure OpenAI.
Databricks	LLM via Databricks Foundation Model APIs / Genie.	Wired through the Databricks workspace integration.
Datahub Private AI (Granite-4)	Platform-hosted private model. Credentials live in Azure Key Vault, not the tenant DB. License-gated.	Tenants can start a 30-day trial directly from the banner.

Why one bad key doesn't break the list

Provider keys are Fernet-encrypted with a secret derived from JWT_SECRET. If that secret rotates, older keys become undecryptable and raise DecryptionError on load — the repository catches it and skips just that row with a warning, so a single corrupted key never breaks the provider lookup or the rest of the list.

Fallback¶

If a code path needs a provider the tenant hasn't configured (e.g. a service hardcoded openai but only Anthropic is available), the platform falls back transparently. The model name is rewritten to the alternate provider's default and the request is retried. A structured provider_key_fallback log helps admins see the drift.

flowchart LR
    REQ["Call needs provider 'openai'"] --> LOOK{"Active key<br/>for provider?"}
    LOOK -->|yes| USE["Use most-recent key<br/>corrupted rows skipped"]
    LOOK -->|no| FB["Transparent fallback<br/>rewrite model to alternate<br/>provider default · retry"]
    FB --> LOG["Log provider_key_fallback"]
    USE --> CALL["Provider call"]
    LOG --> CALL
    FB -.no key anywhere.-> ERR["503 No active API key"]
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class FB accent

Figure 2 — Provider-key resolution. The most-recent active key wins; a missing provider triggers a transparent rewrite-and-retry against an alternate, logged as provider_key_fallback. Only a total absence of keys surfaces a 503.

Azure OpenAI¶

Organisations that run OpenAI models on their own Azure resource add an Azure OpenAI key. Besides the API key, the form asks for:

Field	What to fill in
Endpoint	The resource's address, e.g. `https://contoso.openai.azure.com`. A v1 address (`…/openai/v1`) or one ending in `/openai` works too.
Default deployment	The deployment used when Model Settings names none, e.g. `gpt-4o-mini`.
API version	Leave empty to use Azure's v1 API. Fill in a dated version (`2024-10-21`, `2025-04-01-preview`) only if your resource needs one.

Datahub sends the key to the endpoint, so only Azure hosts are accepted, over https on the standard port: *.openai.azure.com, *.cognitiveservices.azure.com, *.services.ai.azure.com, *.models.ai.azure.com and API Management gateways (*.azure-api.net). A custom domain, an address with a user name, query or port, and plain http are refused. The platform health check probes the resource with the key, so a wrong endpoint or key shows up there.

In Model settings, choosing Azure OpenAI turns the model field into a deployment field: type the name of a deployment on your resource, or leave it empty to use the key's default deployment. The same goes for a custom agent's model. Name deployments after their model (gpt-4o, gpt-4o-mini-prod, not chat-1): Datahub reads the model family from the name to send the request in the right shape.

What runs on Azure OpenAI: HERC, the built-in and custom agents, Insights AI, glossary enrichment, catalog document processing and the semantic layer. Not yet:

The Brain (Organisation DNA) does not build on Azure OpenAI. When the default model is on Azure, the Brain says so and asks for an OpenAI or Anthropic model; the embeddings it needs still use an OpenAI key.
Transcription (voice input) runs on OpenAI's Whisper and needs an OpenAI key. Without one it says that transcription does not run on Azure OpenAI yet.

Fallback can leave Azure

The fallback uses another provider's active key when the configured provider has none, and HERC also moves a turn to one when Azure keeps failing. An organisation that chose Azure OpenAI to keep data in its own tenant should keep only Azure keys active: with an OpenAI or Anthropic key active too, a failing, deactivated or missing Azure key sends HERC's turns to that provider instead. Provider Keys warns while an Azure key and another provider's key are both active, and names the other providers.

Datahub Private AI¶

The platform-hosted Granite-4 model on Datahub-managed hardware. Three tiers:

Tier	Pricing	Where the model runs
Business	From €100 / month	Shared sovereign Granite-4 instance. 30-day trial available.
Enterprise	Base fee + per-token usage	Dedicated / siloed Granite-4 instance per organisation.
Enterprise+	Custom	Granite-4 + Datahub management software deployed inside your environment.

Trial flow is self-service: click Start trial in the locked banner, fill in name / email / company, the license server activates a signed trial license, and the tenant becomes entitled immediately. The trial is hard-capped at a token limit and a 30-day window; the admin cannot raise the cap during trial.

For paid plans, monthly token usage is metered and visible at /ai/provider-keys (per-agent breakdown, hard-limit toggle, warn threshold). Hitting the hard limit returns 429 to the caller, and the Brain does not rebuild on Private AI until the limit is raised or the month ends (see Organization DNA). The usage shown counts the agents' calls and the Brain's builds that read your content with Private AI, listed as brain in the per-agent breakdown (a build that failed part way counts what it used). Embeddings are not counted: they run on your own OpenAI key.

Model settings — `/ai/model-settings`¶

Per-agent overrides for which model + what parameters to use:

Parameter	What it controls
Provider + model name	E.g. `openai / gpt-4o-mini`, `anthropic / claude-haiku-4`, `datahub_private / granite4@dev`. On Azure OpenAI the model is a deployment name (`azure_openai / gpt-4o-prod`); empty uses the key's default deployment.
Temperature, top-p	Determinism vs creativity.
Max tokens	Hard ceiling per response.
Frequency / presence penalty	Repetition penalties. (Auto-stripped for providers that don't support them, e.g. Anthropic, Granite.)

Two layers:

Global default — the model used when an agent has no override.
Per-agent override — e.g. give the metric agent a stronger model and the catalog agent a faster one.

flowchart LR
    START["Agent call"] --> A{"Per-agent<br/>override set?"}
    A -->|yes| AV["Use agent model + params"]
    A -->|no| G{"Global default<br/>configured?"}
    G -->|yes| GV["Use global model + params"]
    G -->|no| HD["Hardcoded default<br/>openai / gpt-5.4-nano"]
    AV --> OUT(["Resolved model config"])
    GV --> OUT
    HD --> OUT
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class OUT accent

Figure 3 — Model resolution precedence: a per-agent override wins over the global default, which wins over the hardcoded fallback. Datahub Private AI is filtered out of the dropdowns unless the tenant is entitled.

Result limits¶

How many rows Genie and a SQL warehouse hand an agent, and how long each is waited on. They hold for the whole organisation. Leave a field empty to use the default. An administrator who manages provider keys sets them; a value outside the range is refused, and the message names the range.

Limit	Default	Range
Rows per Genie answer	50	1 – 1,000
Wait for Genie	90 s	1 – 300 s
Rows per SQL query (Databricks and Fabric)	50	1 – 1,000
Wait for a Databricks SQL query	65 s	20 – 300 s

When a result has more rows than the limit, the agent says so, for example "Showing 50 of 312 rows".
A Databricks query runs for up to 15 seconds less than its wait, so its answer arrives in time. A Fabric query has no wait limit yet.
An MCP client such as Claude or Copilot may stop waiting after about 100 seconds on its own, whatever the wait is set to.
Every change is written to the AI audit trail with who made it and the old and new value.

The page lists every built-in agent, and each custom agent with the model its live version uses. A custom agent's model is shown read-only here: it is part of the agent's version, so you change it in the agent's builder and it goes live through review (see Custom agents).

Custom agents — `/admin/ai/agents`¶

A custom agent is an agent your organisation builds for its own data, next to the built-in ones. The Agents tab lists them with their state, and New agent opens the builder. The full guide, with the developer contracts and a worked three-agent example, is Custom agents. Each agent has a fixed identifier and a kind:

Kind	What it does
Specialist	Answers questions in one area, with its own tools and data.
Router	Reads a question and hands it to the agent that should answer it.
Reviewer	Checks another agent's answer before it reaches the person who asked.

The builder has one section per part of the agent:

Section	What you set
Identity	Its name, its identifier (fixed once created), when to use it, and its kind.
Purpose	The instructions it follows on every question. The platform adds its own rules around them: the asker's language, when to refuse, and citing evidence.
Tools	What it may call, from the tool catalog. Tools that change data are marked.
Consult	Other agents it may ask for help. An agent may not consult itself, and a loop of agents consulting each other is refused.
Data scope	The Genie spaces, integrations and tables it may read. A list left empty does not narrow what it reads. A reviewer reads only what its own scope and the scope of the agent it checks both allow.
Limits	Rows per query, the wait per query and the number of tool rounds. Left empty, the organisation's result limits apply.
Model	Its provider, model and parameters. Left empty, the global default model applies.
Access	Whether it runs with the asker's own access or with its own, and, when it runs with its own, who may call it: role groups, and business roles shown as the role's name followed by (business role). Naming a business role lets everyone who holds it call the agent.
Change summary	What this version changes, for the reviewer.

If a list the builder needs could not be loaded, such as the tool catalog or the Genie spaces, the builder says so with Try again. It never shows an empty list as if nothing exists.

From draft to live¶

Every change to an agent is a version, and only a reviewer puts a version live:

Create saves the first draft.
Submit for review sends the draft to a reviewer. It refuses a draft that changes nothing against the live version, a model provider the platform does not know, and a model name without a provider, as well as a consult loop and tools, agents, Genie spaces, integrations or role groups that no longer exist. While it is under review the agent cannot be edited; Withdraw takes it back to draft.
The reviewer approves or rejects it in their task, which opens that version. The person who submitted a version never approves it. A rejected version goes back to draft.
When a version is approved it goes live, and the version that was live before it is kept in the history.
A version that runs with its own access needs two approvals. The agent approver (aimodule.agents.approve) reviews it first. Their approval does not put it live: it sends the version on to a second review by the data owner, someone who may approve integrations (integrations.approve). Only the data owner's approval puts it live. The agent approver is not asked for the second review and may not approve it, unless nobody else may; they may still reject it. A rejection or withdraw at either review takes the version back to draft, and it starts again at the agent approver.
The agent's own account follows the live version. An agent that runs with its own access acts through an account of its own, with the roles of its role group; role-group admins decide those roles. The account is created switched off with the first draft that runs with its own access, so its role group can be given roles before review; it acts only once such a version goes live. It is switched off again when a version that runs with the asker's access goes live, or the agent is archived. An agent that already ran with its own access before these accounts existed gets its account when Datahub next starts, with no roles: until a role-group admin gives its role group roles, it has no access of its own.
Editing a live agent starts a new draft. The live version keeps answering until the new one is approved.
Archive takes an agent out of use and withdraws a review in progress. Before you confirm, it names the agents that consult this one: they lose it as a consult target. Restore brings it back. An agent is never deleted.
A reviewer agent a specialist relies on stays a reviewer. While a live specialist lists a live reviewer agent in Consult, and no other live reviewer, that reviewer cannot be archived, and a new version that makes it another kind is not approved. The refusal names the specialists. Archive them first, or approve a new version of each that no longer lists the reviewer.

Activate is shown but greyed out: approval is the only way a version goes live. Open any version from the history to read it next to the live one.

While a version runs with its own access, the builder and the version view say what it exposes: which role groups may call it and which data they can see through it, including data they can't open themselves; the roles the agent acts with; and that it goes live only after both approvals.

stateDiagram-v2
    state "Data owner review" as DataOwnerReview
    [*] --> Draft: Create
    Draft --> UnderReview: Submit for review
    UnderReview --> Draft: Withdraw or reject
    UnderReview --> Live: Reviewer approves (asker's access)
    UnderReview --> DataOwnerReview: Agent approver approves (own access)
    DataOwnerReview --> Draft: Withdraw or reject
    DataOwnerReview --> Live: Data owner approves
    Live --> Superseded: Next version approved
    Live --> Draft: Edit starts a new draft
    classDef accent fill:#ff8400,stroke:#e67700,color:#fff
    class Live accent

Figure 4 — The life of a custom agent version. Editing a live agent starts a new draft while the live version keeps answering; only a reviewer's approval puts a version live, and one that runs with its own access needs the data owner's approval as well.

Try a draft before you send it for review¶

Try, in the builder's header, lets you ask the agent's saved draft a question and see how it answers before a reviewer does. It opens a Test run in the side panel: type a question and choose Run test. The answer shows as the same step the chat shows when HERC asks a specialist, live while it runs: the tools the draft called, the SQL it ran with the number of rows it found (never the rows themselves), and its answer.

Who sees Try. Try is shown on a saved agent that is not archived, to someone who may build agents (aimodule.agents.manage) and run them (aimodule.agent.run). Someone who may build agents but not run them sees Try greyed out, with a note naming the right they are missing. CustomAgents.Editor holds both, so its members can try their drafts. Platform.Contributor builds agents but does not hold aimodule.agent.run, so a builder who has only that tier sees Try greyed out until they also join a role group that holds it, such as CustomAgents.Editor or AI.Editor.
The saved draft, nothing else. The test run asks the draft as it was last saved. While the builder has unsaved changes, Run test waits until you save. It never tries the live version: on a live agent, change something and save it to start a draft. A version under review is fixed and can't be tried; Withdraw it to change it and try it again.
Always your own access. The test run uses your own access, even when the draft's Access is set to run with its own. It sees only what you may see, under your role's tool policy. A version nobody has approved never runs with access of its own. Within that, the draft's own Data scope and limits hold, not the live version's, so you see what the version you send for review would read.
Nothing is saved to a conversation. The question and the answer are not stored as a conversation and do not appear in your chat history.
Usage counts as usual. The test run's AI usage is recorded like any other run's. Your question and the draft's answer pass the guardrails as a chat turn's would. Each run is written to the AI audit log as a request to the agent, with the draft version that answered and you as the person who asked.

A test run asks the draft the way HERC would

The draft is asked as HERC would ask a specialist (see How HERC asks a specialist), so it only looks things up: tools that change data are not offered, and a draft whose every tool changes data is not asked. HERC itself never asks a draft; a version reaches the chat only once a reviewer approves it. Two things a test run does not do yet: a reviewer in the draft's Consult list does not check its answer, and a draft router or reviewer is asked your question as if it were a specialist.

How a chat turn reaches them¶

A live custom agent answers through HERC: HERC asks it and writes the reply itself, as it does with a built-in specialist (see How HERC asks a specialist). A turn reaches your organisation's agents in three ways:

Picking or naming one. The chat's @ menu lists your organisation's live specialists you may ask, next to the built-in agents: those that run with your own access, and those that run with their own when you are in one of the role groups, or hold one of the business roles, that may call them. A specialist with its own access is offered only to those people, by the menu, by HERC and by the router alike. Pick one, or name it in your message by its name or identifier, and HERC asks it first. Routers and reviewers are not in the menu: the platform asks them itself. If the agent you picked can't be consulted for you when you send, a step says "The agent you picked, 'identifier', can't be consulted for you; this answer does not come from it." HERC says so in one sentence and answers as if you had picked nothing. That happens when the agent is no longer live, when you may not call it, or when your role's tool policy leaves it no tool to answer with; neither says which.
A live router. When a router has a live version, HERC asks it before answering, on every turn except one where you picked an agent or where your message answers a question a specialist asked you. The router reads your message, and the one before it so that a short follow-up reads as what it follows up on. It chooses one of the specialists you can pick, or says that none covers the question. HERC then asks the chosen specialist first. When none covers it, HERC answers with its own tools or a built-in specialist, or says that nothing covers it; a message that follows up on an earlier answer from a custom agent still goes back to that agent. While a router is live, naming an agent in your message does not reach that agent directly: the router reads the name. If the router could not be asked, failed, or gave an answer that could not be read, HERC chooses for itself. An organisation has one live router. A second router is refused when it would go live, at approval, and when you restore one while another is live; the refusal names the live router, so archive that one first. If two routers were already live before this rule, only the first by identifier is asked.
A reviewer on a specialist. When a specialist's Consult list includes a live reviewer, that reviewer checks every answer the specialist gives before HERC reads it, against your question, the task, the answer and its evidence. HERC never presents a rejected answer as fact. When the reviewer's verdict could not be read, the answer counts as not checked, never as approved. So does the answer of a specialist whose Consult list names a reviewer that is not live (archived, or not approved yet) and no live one.

In the chat, the router's and the reviewer's requests show as steps like HERC's own, marked Router or Reviewer. The router's step says which agent it chose, that no agent covers the question, or that HERC chose for itself; when the router was not asked or failed, its step also says why. The reviewer's step shows Approved, Rejected or Not checked, and a rejection keeps its reasons in sight. The specialist's own step says who checked its answer and how.

Without a live router or reviewer, custom agents are reached only by picking or naming them. Without any custom agents, the chat works as before.

sequenceDiagram
    autonumber
    participant U as You
    participant H as HERC
    participant R as Router
    participant S as Specialist
    participant V as Reviewer
    U->>H: Question
    H->>R: Which agent covers this?
    alt a specialist covers it
        R-->>H: The specialist
        H->>S: Task + evidence
        S-->>H: Answer + SQL + rows
        H->>V: Question, task, answer, evidence
        alt approved
            V-->>H: Approved
            H-->>U: One answer, citing the query
        else rejected
            V-->>H: Rejected, with reasons
            H-->>U: A reply that does not present it as fact
        end
    else none covers it, or no usable answer
        R-->>H: None
        H-->>U: Answers with its own tools or a built-in specialist
    end

Figure 5 — A turn with a live router and a specialist that has a reviewer. The answer reaches you only through HERC, and a rejected answer is never presented as fact.

Drafting alerts¶

HERC's triage and Databricks agents can draft an alert and take it as far as review: create_alert_rule, update_alert_rule, test_alert_rule, preview_alert_query, preview_alert_message, list_alert_connections, submit_alert_rule_for_review and withdraw_alert_rule_review. The Insights agent can start one from a dashboard card with create_alert_from_card. All of them stop at a draft: a person approves it. See Logic Engine and MCP governance.

Editing reports¶

The Insights agent, and an AI assistant connected over MCP, can read an Insights report and change it on its draft: get_insight_dashboard, update_insight_dashboard (pages, page filters, name), move_insight_card, archive_insight_card, archive_insight_dashboard_page, restore_insight_dashboard_page and the card tools, which set a table's column formats and grouping too. It then sends the draft with submit_insight_dashboard_for_review. Readers see the published version until a person in the reviewing group approves the draft; no tool approves it. A page filter's options query gets a card's checks: read-only SQL, on tables the agent may read. See Insights.

Guardrails — `/ai/guardrails`¶

Pluggable content-safety pipeline that runs before the LLM sees the prompt and after the LLM responds. Rules are direction-scoped (input, output, or both) and optionally agent-scoped.

Rule types:

Type	What it does	Default?
PII redact	Detects + masks email, phone, SSN, credit card, IP via regex.	Yes (input + output).
Regex block	Block on a custom regex.	Yes (input).
Topic block	Reject prompts touching a configured topic list.	No.
Disclaimer	Prepend / append a fixed disclaimer.	Yes (output).
Prompt injection	Heuristic regex with three sensitivity tiers.	Yes (input).
Content moderation	OpenAI Moderation API (free, fail-open). Uses the platform moderation key (or the tenant's OpenAI key as fallback).	Yes.
Secret detection	AWS keys, API keys, bearer tokens, private keys, connection strings, JWTs.	Yes.

Six rules ship out of the box; tenants can add more. Every violation is logged as a span attribute (datahub.guardrail_violations) and visible in the trace detail.

Dry-run before you activate

The Test button on each rule lets you push sample text through the pipeline and see exactly what gets redacted or blocked — before the rule ever touches a live prompt.

Observability — `/ai/observability`¶

The native trace browser, backed by Arize Phoenix.

Every agent call is automatically traced via OpenTelemetry. The trace dashboard shows:

Metric	What it tells you
Trace count, error rate, P50 / P99 latency	Operational health.
Token usage over time	Spend pattern.
Per-agent breakdown	Which agents are called most.
Estimated cost	Blended $ / 1M tokens (configurable).

Reading a date range. Pick a range in the table's Time range filter: the last 24 hours, 7, 14 or 30 days, All kept traces, or your own days (no further back than the 60 days kept). The page starts on the last 7 days. It loads the newest 100 traces of the range (500 spans in the All spans view); Load more below the table reads the next, older ones, until the line under the table says the whole range is loaded. The figures and charts are worked out from what is loaded, and the note above them says so: "the 100 newest traces loaded" while the range has more, "all 37 traces in this range" once it is all loaded. While only the first page is loaded it refreshes every 15 seconds; once you load more, it holds still until you press Refresh. The range and the kind, status and evaluation filters are not saved in a table view. The search and the kind, status and evaluation filters look only at the traces loaded so far: while the range has more, the line under the table says "Filtering the 100 loaded traces; load more to search older ones", and the note above the figures says how many of the loaded ones match. If Phoenix does not answer, a red message says the list could not be read (never that the whole range is loaded): press Refresh. If the list changed under the page you were on, the message says so and Refresh starts again from the newest trace. When more traces started at a single moment than the list can page past, the line under the table says the list is not the whole range: pick a range that ends before that moment to see older ones. A group of traces with more spans than the page reads at once is still listed, but an amber message says their tokens, input and output may be incomplete; open a trace to read its own spans.

Click any trace to see:

The span waterfall — orchestrator → tool calls → LLM → guardrails → response.
Attributes (model, provider, agent name, conversation id, hashed user id, guardrail violations).
The actual prompt + response (with PII already redacted by the guardrails).
Evaluations from Phoenix (auto-eval rules that score relevance, faithfulness, etc.).
Annotations — manual feedback you and your team add to traces (label / score / freeform). Define schemas at Observability → Annotation configs.

Traces are kept for 60 days

Deployed, Phoenix runs as its own service and keeps its traces in the platform's database for 60 days, so they survive updates and restarts. Older traces are removed once a week, so the oldest stored trace can be up to 67 days old. The page reads a date range: the last 7 days to start with, back to the oldest day Phoenix keeps (All kept traces); a range that starts earlier is refused. Phoenix is reachable only from inside the platform, behind its own login (its own UI is not exposed: this page is the way in), and the platform signs in with a key it keeps in the key vault. Only in development, when no Phoenix is reachable at DATAHUB_PHOENIX_COLLECTOR_ENDPOINT, does the platform start one in-process; deployed, it keeps retrying the configured Phoenix, and traces from a turn while Phoenix is down are lost.

Experiments — `/ai/experiments`¶

Offline evaluation suite for agents. The flow:

Create a dataset of prompts (with optional gold-standard outputs). Bulk-add or import CSV.
Create an experiment: pick the dataset and the agent to test.
Execute — the platform runs every example through the agent (using the same MCP tool filtering as production), capturing per-example latency, token counts, and output.
Compare runs across agents / model settings / prompt versions.

Caps: max 100 examples per run.

Use it to:

A / B test model upgrades before flipping the default.
Catch regressions when a system prompt changes.
Validate that guardrails don't over-block legitimate queries.

How HERC edits SQL¶

HERC never changes a query on its own. When you ask it to write or fix SQL, it proposes the SQL and you decide.

HERC checks the SQL before you see it: it must only read data, every table alias must exist where it is used (for example an alias defined inside a WITH block cannot be used after it), and, when your organisation allows HERC to read data from the connection, it runs the query once to prove it works. SQL that fails a check goes back to HERC to fix; you never see it.
The proposal appears under HERC's reply once HERC has finished its turn — never while it is still working: a short summary, how it was checked, and the full SQL with a Copy button. A proposal is meant to be the query you use: SQL that only looks at the data (such as SELECT * of a table with a small LIMIT, or DESCRIBE) is refused before you see it. Once you write again, earlier proposals fold to their header; open one to read or apply it. If you stop HERC, or its turn fails, after it proposed SQL, the proposal stays, marked as not yet HERC's conclusion.
Apply puts the SQL into the editor it was written for (a card, a saved query or an alert). Nothing is saved yet: review it in the editor and Save there, which follows the normal review flow. Undo puts back what Apply replaced, as long as you have not edited it since.

If a connection shares only metadata with AI (the default), HERC cannot run SQL on it. It reads the table's columns from the schema instead and still gives you a proposal with Apply. The proposal says whether the SQL could be run; when it says Checked, not run, run it in the editor before you save.

If the connection's warehouse was stopped, it takes a few minutes to start. HERC then tells you it is starting instead of failing; ask again in a few minutes. If you already gave HERC enough to write the SQL, it still proposes it, marked Checked, not run. A query that runs but is slow gets the same treatment: HERC does not keep re-running it, and suggests narrowing it instead.

Use Ask HERC inside the card, saved-query or alert editor so HERC knows which SQL you mean and the proposal can be applied. While that editor stays open, every message you type in the same chat carries the editor's current state too — its SQL (including SQL you applied or changed since), columns and settings, never rows of data — so HERC works from what is in the editor now. A message that plainly asks about something else, such as the glossary, still goes to the right specialist. SQL HERC gives in the general chat can be copied, and applied when exactly one SQL editor is open.

How HERC asks a specialist¶

Some questions need more than one area, for example a glossary definition together with a figure from the warehouse. For these, HERC can ask the right specialist, use what comes back, and write one answer itself, without handing the conversation over.

flowchart LR
    U["You"] -->|question| H(["HERC"])
    H -->|task + evidence| S["Specialist<br/>fresh context, looks things up"]
    S -->|answer + SQL + rows| H
    H -->|one answer, citing the query| U
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class H accent

Figure 6 — HERC keeps the turn. The specialist answers HERC, not you, and HERC writes the reply.

The specialist sees only the task HERC writes for it, never your earlier messages. It works within the same tool access as you.
A specialist asked this way only looks things up. It cannot change anything, hand the conversation over or ask you a question. For a change, HERC hands the conversation over to the specialist instead, so you talk to it directly.
When HERC quotes a figure from the warehouse, it can say which query it came from.
A turn asks at most 6 specialists. When too little of the turn's time is left, HERC says so instead of asking.
In the chat, each request shows as a step under HERC's reply: which specialist was asked, the question HERC put to it, whether it answered and how long it took. Open the step to see what the specialist did, the SQL it ran with the number of rows it found (never the rows themselves), and its answer. A request that was not made, or that failed, says why. While HERC works you see the specialist's progress live, and the steps are still there when you reopen the conversation.
Every request is recorded in the AI audit log: who asked (by name), which specialist, the outcome and how long it took (never the task text itself). A request that was refused or failed says why in plain words, with the facts behind it, such as how many seconds were left or how many levels deep requests may nest. A record names the conversation it happened in and, when one specialist asked another, the request that led to it, so a chain can be followed back to the person who asked. The step in the chat names its record, so HERC can look up why an earlier request in the same conversation was refused. You can read the consults made on your own questions through the API or by asking HERC which specialists it consulted and why one was refused; an administrator with the audit role can read everyone's. There is no screen to browse them yet.

How the AI platform fits with the rest¶

Feature	Depends on
HERC chat	At least one provider key + a model resolution (global or per-agent).
Metric suggestions, dashboard generation, term enrichment	Same.
Transcription (Whisper)	OpenAI provider key.
Genie chat on dashboards	Databricks workspace credentials (proxied — no AI provider key needed).
Conversation persistence	Built-in — every chat is stored per-user with optional summary on archive.
Conversation language	Driven by the user's UI locale; the platform prepends `[LANGUAGE: en\|nl]` to every prompt to enforce adherence.

Roles¶

Role	What you can do
`aimodule.providerkeys.read`	View provider keys, model settings, guardrails, observability, experiments.
`aimodule.providerkeys.write`	Manage provider keys, model settings, guardrails.
`aimodule.agent.run`	Run conversations (HERC chat).
`aimodule.conversations.read / write`	Read / manage conversations.
`aimodule.transcription.run`	Use the Transcription module.
`aimodule.agents.read`	View custom agents and their versions.
`aimodule.agents.manage`	Create custom agents, edit and submit their drafts, withdraw a review, archive and restore. Building an agent needs `aimodule.agents.read` as well; trying a draft needs `aimodule.agent.run` too, which `CustomAgents.Editor` holds and `Platform.Contributor` does not.
`aimodule.agents.approve`	Approve or reject a custom agent version in review. A version that runs with its own access then needs a data owner's approval as well (`integrations.approve`).

Bundled into role groups: Administrator and AI.Editor get all AI roles. AI.Viewer is read-only.

The custom agent roles are the exception: AI.Editor and AI.Viewer do not hold them. Administrator holds all three. CustomAgents.Viewer views agents, CustomAgents.Editor builds and tries them (it also holds aimodule.agent.run), and CustomAgents.Approver views and approves them but does not build them. The platform tiers hold them by verb: Platform.Viewer reads, Platform.Contributor also builds, and Platform.Approver also approves.

Limitations¶

Limit	Why	Workaround
Provider keys are tenant-scoped, not user-scoped.	Most teams want one shared budget.	Use observability + guardrails to track per-user usage.
Granite-4 (Private AI) is license-gated.	Commercial offering.	Start a 30-day trial from the banner; upgrade to Business / Enterprise / Enterprise+.
Phoenix dashboards are best on a fresh page load.	OTel batching.	Refresh after a recent call if you don't see it.
Experiments cap at 100 examples per run.	Cost guardrail.	Split into multiple datasets.
Conversation history is trimmed for classification + follow-ups.	Token budget.	Long-running threads still answer; classifier just sees the recent slice.
A HERC turn runs for at most 10 minutes.	Long tool calls are kept alive, but a turn still needs an upper bound.	A turn that takes longer ends with "the request took too long"; split the question into smaller steps.

Audit & compliance¶

Question	Where to look
"Where do prompts go?"	The provider configured for the agent — visible per call in `/ai/observability`. With Datahub Private AI, prompts stay on platform infrastructure.
"Who asked the AI what?"	Per-conversation (and per-message) trace tied to the user JWT. Visible in observability + the conversations API.
"Did the AI see PII it shouldn't?"	Guardrails redact PII before the prompt leaves the platform. Violations are logged on the trace.
"How much did this cost?"	Observability → estimated cost (blended) + per-agent token breakdown. For Private AI, the metered ledger lives on the provider-keys page.
"Are conversations encrypted at rest?"	Yes — provider keys are Fernet-encrypted; conversations are PostgreSQL rows.
"Can I disable a model after the fact?"	Yes — delete the provider key + remove the per-agent override. The platform falls back transparently.

What a chat error means¶

When a HERC turn fails, the assistant shows which failure it was and who can fix it, in place of a reply. The message is the same in Dutch and English, and the ones naming an administrator action are pointing at a page in this guide.

The failure is saved with the conversation, so it is still there after a reload. On the latest turn it has a Retry button that sends the same question again. Retry is offered for every row below except two, where the same question fails the same way: a conversation too long for the model, and a question the content filter blocked. Where an administrator has to act first, Retry works once they have.

What the user is told	What actually happened	Who fixes it, and where
"the organisation's account is out of credit"	The AI provider refused the request for billing reasons. Retrying will not help until the account is topped up.	An administrator tops up the account behind the key — Provider keys.
"refused our API key, which usually means it was revoked or rotated"	The provider rejected the credential itself.	An administrator sets a working key — Provider keys.
"the AI model … is not available on the provider account"	The configured model does not exist for this key.	An administrator selects another model — Model settings.
"this conversation has grown longer than the AI model can take"	The thread outgrew the model's context window. Nothing is lost.	The user starts a new conversation or asks a shorter question.
"the AI provider's content filter blocked this request"	The provider refused on content grounds, not Datahub's guardrails (those answer in their own words — see Guardrails).	The user rephrases.
"the AI provider hasn't been set up yet"	No API key is configured for the provider at all.	An administrator adds one, Provider keys. Common on a fresh environment.
"the AI token quota for this organisation has been used up"	The Private AI token quota is spent. Retrying will not help until the quota is raised or resets.	An administrator raises the quota or waits for the next period; usage is under Observability.
"The connection was closed before HERC finished, so this answer may be incomplete"	The connection went quiet part-way through a turn (a network drop or a proxy closing it), so the reply may be cut short. The message may still have been processed.	The user checks the conversation, then sends it again if needed. If it keeps happening, an administrator checks the network path between the browser and Datahub.
"Something unexpected went wrong" plus a Reference	The failure could not be classified. The reference is the run id the exception was logged under.	Pass the reference to whoever reads the logs — it identifies the exact run without a reproduction.

Rate limiting, timeouts and provider outages keep their own messages and are all "wait, then Retry" cases; only the rows above need a person to do something.

Who hears about it¶

The rows that need an administrator reach one without the user passing them on. When a chat turn fails on one of them, whoever holds aimodule.providerkeys.write gets a notification in their bell with the message the user saw, at most once an hour per error. If nobody holds that role, the Administrator business role is told, and failing that the Administrator role group.

The AI provider card on the AI hub counts the agents whose last run in the past day failed this way. An agent leaves the count when a run succeeds, or after a day without a run. Only chat turns count for now.

The Prompt regimes card on the same hub counts approved, active regimes bound to a business role nobody holds, which therefore shape nobody's answers. Whoever holds regimes.admin is told, or the Administrator business role when nobody does.

Queuing follow-ups while HERC works¶

You can keep typing while HERC thinks, calls tools or writes its answer. Pressing Enter while HERC is busy puts the message in a queue above the message box instead of interrupting the run. HERC sends the queued messages one at a time, each after the previous answer is complete, and only into the conversation they were typed in.

Situation	What the queue does
HERC is busy	New messages join the queue, up to 10. When it is full, the message stays in the box until you remove a queued one.
You press Stop, or a run fails	The queue pauses. Resume sends the next message; Clear empties the queue. A message whose run failed goes back to the front, so Resume retries it.
HERC asks you something	The queue waits for your answer. Send as reply sends the next queued message as that answer, only when you choose it. For a clarifying question with reply options the wait is certain. A question HERC asks in plain text is recognised by its wording, so that is best effort: a question worded differently can be missed, and then the next message goes out.
You switch conversation, start a new chat or sign out	The queue is dropped, and a notice says how many messages were discarded.
You reload the page	The queue comes back paused, for the same conversation in the same browser tab.

Troubleshooting¶

Symptom	Likely cause	Fix
`503 No active API key found for provider 'openai'`	Tenant has no OpenAI key + service hardcoded `openai`.	The fallback should engage automatically; if not, add an OpenAI key.
`429 Too Many Requests` from Private AI, or the Brain page says Private AI limit reached	Hard token limit hit (or trial cap).	Wait for the next month rollover, raise the limit, or upgrade.
Trace doesn't show in observability	Phoenix not enabled, unreachable, or refusing the platform's key.	Check `DATAHUB_PHOENIX_ENABLED`. Platform health shows Phoenix as degraded with "Phoenix refused the app's API key" when `DATAHUB_PHOENIX_API_KEY` is not Phoenix's admin secret. In development, the in-process Phoenix should auto-launch.
Guardrail blocks a legitimate prompt	Topic / regex rule too aggressive.	Test rule from the guardrails page; tune sensitivity.
HERC says "The agent you picked … can't be consulted for you"	The custom agent is not live (archived, or not approved yet), you may not call it, or your role's tool policy leaves it no tool to answer with.	Get its version approved or restore it, ask to be added under its Who may call it, or ask an administrator to check your role's tool policy. HERC answers without it meanwhile. See Custom agents.
HERC keeps responding in the wrong language	Locale not resolving from JWT.	Check user account locale setting; the language directive prefixes every prompt.