HERC ↔ Databricks Genie¶

Ask HERC a natural-language question whose answer lives in a warehouse table, and it hands off to a Databricks Genie space, runs the query under your own Unity Catalog permissions, and weaves the result back into the chat.

This is where HERC reaches past the Datahub catalog into your warehouse — for questions whose answer is a number in a Unity Catalog table (revenue, customers, orders, pipeline, churn), not a definition in the glossary.

AnswersWarehouse-table questions

Runs asYour Databricks identity · per-user UC RLS

Result50 rows · 90 s · admin-set

SyncManual · idempotent

When to reach for this¶

Answer a warehouse-data question

Sales, finance, marketing funnels — anything living in your Unity Catalog tables. "Top 10 customers by revenue last quarter?"
Respect each user's UC permissions

Genie queries run under the calling user's Databricks identity via oauth_u2m. Users see only the data they'd see in the Databricks UI themselves.
Stay bounded to a space's tables

Genie is not a general SQL editor — it shines when a space's scope is narrow (10-20 well-described tables).
Iterate with follow-ups

"Now split by region" re-uses the same Genie conversation, so the SQL context carries over — far more accurate than starting fresh.

When not to reach for the Databricks path

Governance metadata — which terms exist, who owns an asset, what's in a data product → use HERC's regular catalog / glossary / contracts agents. Much faster than a Genie round-trip.
Detail beyond the row limit → HERC returns at most 50 rows by default so the chat stays readable, and says "Showing 50 of 312 rows" when there are more. An administrator sets the limit (up to 1,000) under Model settings → Result limits. For larger sets, open the space in Databricks and run the SQL Genie suggested.
Modifying data → Genie is read-only, and so is HERC's integration with it.

How HERC routes a Databricks question¶

HERC decides whether a turn is a metadata question (answered from the Datahub catalog) or a warehouse-data question (answered by Genie). Warehouse questions go to the Databricks data specialist, which lists the registered spaces, picks the best-matching one, and asks it — under your identity.

flowchart LR
    Q["Your question<br/>EN or NL"] --> R{"HERC router<br/>catalog or warehouse?"}
    R -->|"metadata: terms · owners · products"| CAT["Catalog / glossary /<br/>contracts agents"]
    R -->|"warehouse data: revenue · churn · funnels"| SPEC["Databricks data specialist<br/>list spaces → pick best"]
    SPEC --> GS(["Genie space<br/>bounded table scope"])
    GS --> DBX[("Databricks<br/>runs as you · UC row-level security")]
    DBX --> ANS["Answer woven into chat<br/>row limit · follow-ups"]
    ANS -.streamed.-> Q
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class GS accent

Figure 1 — A metadata question is answered from the Datahub catalog; a warehouse-data question is routed to a Genie space, executed against Databricks under your own Unity Catalog permissions, then folded back into the chat with a sample of the result.

Why teams reach for this

You ask in plain language, the answer respects the exact same Unity Catalog permissions you already have, and every follow-up re-uses the Genie conversation — so "now slice it by region" keeps the previous quarter's filter and the revenue measure. Let HERC chain follow-ups whenever the user is iterating.

What it looks like for the user¶

You: What were our top 10 customers by revenue last quarter?

HERC: I asked the Sales Analytics Genie space. Acme Corp led at €842 k, followed by Globex (€640 k) and Initech (€515 k). Here's the top 10:

Customer Revenue (€)

Acme Corp 842,000

Globex 640,000

… …

Want me to compare this to the previous quarter, or slice it by region?

A follow-up like "now split by region" re-uses the same Genie conversation, so the SQL context (the previous quarter's filter, the revenue measure) carries over.

How HERC decides where a question goes¶

The router uses two signals to send a question to Genie:

Signal	When it fires	Where it routes
Fast keyword routing	Phrases like "ask databricks", "from databricks", "in the warehouse", "which genie space" — and the Dutch equivalents uit databricks, in onze data, vraag aan databricks.	Straight to the Databricks data specialist — no LLM classifier call.
LLM orchestration	Ambiguous questions: a bare "show me top customers by revenue" could go to Insights (edits dashboard cards) or Databricks data (queries the warehouse).	The orchestration layer's LLM resolves which agent owns the turn from the surrounding conversation.

The generalist can still reach Genie

The Databricks data specialist's tools are also available to the generalist datahub agent. So even when routing picks the generalist, HERC can still reach a Genie space.

Concepts¶

Concept	What it is
Genie space	A Databricks-authored bundle of tables, descriptions, sample questions, and instructions. Datahub registers spaces — it does not create or edit them in Databricks.
Registry	Datahub's local list of which spaces HERC is allowed to use, who has curated them, and what they cover.
Description override	An admin-authored description HERC's LLM reads when choosing a space. Preserved across re-syncs.
Active toggle	Off → the space is hidden from HERC entirely: it won't appear in `list_genie_spaces` and can't be the target of an `ask`. Toggling off does not delete the row.

Setup (admin)¶

The integration has two prerequisites.

Prereq	Where	Why
Databricks workspace on U2M OAuth	Admin → Connected apps → Databricks	Genie queries run under the calling user's Databricks identity via `oauth_u2m` — the same flow embedded dashboards use. See Databricks: per-user authentication (U2M OAuth).
≥ 1 Genie space registered	Admin → Integrations → the Databricks integration → Genie spaces tab → Sync	Spaces are authored in Databricks; Datahub keeps a local registry of which spaces HERC may use, who curated them, and what they cover.

Sync Genie spaces from Databricks¶

Go to Admin → Integrations in Datahub, open the Databricks integration, and select the Genie spaces tab (gated by the databricksgenie.manage role).
Each connected Databricks workspace shows up as a Sync {workspace name} button at the top of the page. Click it.
Datahub queries Databricks for every Genie space accessible to your workspace identity (list_spaces), fetches each one's metadata (get_space with include_serialized_space=True), and upserts them into the registry. The closing toast reports how many spaces were discovered, created (new), and updated (already in the registry — refreshed table list / sample questions).
New spaces appear in the table as Active by default.

flowchart LR
    DBX[("Databricks workspace<br/>Genie spaces authored here")] --> SYNC["Admin clicks Sync<br/>list_spaces + get_space"]
    SYNC --> REG(["Datahub registry<br/>title · tables · sample questions"])
    REG --> TOAST["Toast: discovered ·<br/>created · updated"]
    REG --> ACTIVE["New spaces Active by default"]
    ACTIVE --> CURATE["Admin curates<br/>description override · toggle Active"]
    CURATE --> HERC["HERC sees active spaces only<br/>list_genie_spaces"]
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class REG accent

Figure 2 — The registry lifecycle. Sync discovers spaces in Databricks and upserts them into Datahub's registry; new rows start Active; an admin curates descriptions and activation; HERC only ever sees the active rows.

Why sync is manual

Newly-created Genie spaces should be reviewed by an admin (and optionally given a description override) before HERC starts steering questions at them. Re-syncing is idempotent and safe to run as often as you like — description overrides you've made in Datahub are preserved across syncs, and sync never deletes a space that's no longer visible.

Curate the registry¶

For each space row, click the pencil icon to:

Override the description HERC sees. HERC's LLM picks a space from its title, description, and table list. If two spaces overlap (e.g. "Sales — historical" and "Sales — live"), a short, opinionated description — "Use this space for closed deals only; the live space has open opportunities." — dramatically improves routing accuracy. The override is preserved across re-syncs.
Toggle Active off. Inactive spaces are hidden from HERC entirely. Use this to retire spaces or to silence noisy ones during incident response without deleting them from the registry.

Roles¶

Role	What it grants
`databricksgenie.read`	List Genie spaces from the Datahub registry (used by HERC's tools). Bundled into the DatabricksGenie.Viewer role group, along with the platform baselines `herc.read` and `tasks.read`.
`databricksgenie.manage`	Sync spaces from Databricks and edit registry overrides. Bundled into the DatabricksGenie.Manager role group with the same baselines.

Administrators inherit both. Grant DatabricksGenie.Viewer to anyone who should be able to ask HERC questions backed by Genie; grant DatabricksGenie.Manager to a smaller group of stewards.

What end users need¶

End users do not need access to the admin Genie spaces page. They need the DatabricksGenie.Viewer role (or admin-equivalent), and to have completed the per-user Databricks consent for the workspace whose spaces they're asking about.

flowchart LR
    ASK["You ask a<br/>warehouse question"] --> C{"Consented to this<br/>Databricks workspace?"}
    C -->|"yes"| RUN(["Query runs as you<br/>oauth_u2m · UC RLS"])
    C -->|"no"| MSG["HERC: connect your<br/>Databricks account"]
    MSG --> TASK["High-priority task created<br/>Connect to Databricks"]
    TASK --> CONSENT["You consent once"]
    CONSENT --> RUN
    RUN --> G["Genie answer<br/>woven into chat"]
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class RUN accent

Figure 3 — The per-user consent gate. If you haven't consented for the workspace, HERC asks you to connect and the platform auto-creates a high-priority *Connect to Databricks* task in your inbox. Once you consent, every subsequent Genie-backed question runs as you. See [Databricks: per-user authentication (U2M OAuth)](databricks-u2m-oauth.md).

Limitations and known trade-offs¶

Limit	Why	Workaround
Sync is manual.	Admins review new spaces before HERC steers questions at them.	Click Sync after authoring a space; a nightly auto-sync may come in a later release.
At most 50 rows per answer by default.	Keeps the chat transcript readable and a turn's cost bounded.	An administrator raises it, up to 1,000, under Result limits. For the full set, open the space in Databricks and run the SQL Genie produced.
No write-back — read-only.	Genie, and HERC's integration with it, never mutate.	Make changes directly in Databricks.
No new Genie spaces from Datahub.	Datahub keeps a registry, not a builder.	Author the space in Databricks, then Sync.
A 90-second wait per question by default.	Beyond 90 s the chat-context cost dominates the value of waiting; most answers land in 10-60 s.	An administrator sets the wait, up to 300 s, under Result limits. Run heavy queries interactively in Databricks.

Troubleshooting¶

Symptom	What's happening	What to do
HERC says "No Genie spaces are registered yet."	No rows in the Datahub registry.	Sync from Admin → Integrations → the Databricks integration → Genie spaces tab, then retry.
HERC says "You need to connect your Databricks account first."	The asking user has not yet consented for the workspace that owns the Genie space.	Open the Connect to Databricks task that just appeared in your inbox, or visit `/databricks/consent/{integration-id}`.
HERC says "I don't see a Genie space covering …"	The registry has spaces, but none of their descriptions or tables match the question's domain.	Ask an admin to broaden a space's description override, or to register a new Genie space in Databricks covering that domain and sync it in.
Genie answer takes longer than the wait (90 s by default) and times out	Cold warehouse, pathological query, or large scan.	Retry once (warehouses cache after the first run). For repeated timeouts, simplify the question, run it directly in Databricks, or ask an administrator to raise the Genie wait under Result limits.
Genie returns a `FAILED` status with a SQL error verbatim	Genie could not produce a valid query.	HERC surfaces the error verbatim — the user can rephrase, or an admin can refine the space's description / sample questions in Databricks so future questions land cleaner.