Organisation DNA: the platform's knowledge graph¶

Organisation DNA is Datahub's learned knowledge graph — it reads the platform's active, published governed entities and turns them into the connected understanding that powers conversational AI, semantic search, and exploration.

DNA is the connective tissue between everything Datahub governs and everything its AI does with that knowledge. It is what makes HERC's answers specific to your organisation rather than generic.

EngineMicrosoft GraphRAG

Sources12 governed modules + documents

RefreshWeekly · incremental

PowersHERC · Search · Graph

When to reach for this¶

Explore the connections

Click an entity → see its neighbours → click again → walk the graph and sanity-check how your concepts relate.
Audit coverage & diagnose thin answers

See the entity count, type breakdown, and source-module coverage. If HERC keeps missing things, the run history and index size here reveal a stale or thin brain.
Trigger a re-index after a big change

A bulk import, a new domain, a contract restructure — re-index so the AI catches up.
Run free-text semantic search

Ask "what entities relate to churn?" and get a ranked list of hits across every module.

When you don't need DNA

Making HERC work day-to-day — DNA runs in the background; you never have to touch it.
Browsing the catalog → use Data Catalog.
Adding new terms → use Business Glossary.

What DNA connects¶

DNA sits between everything Datahub governs and everything the AI does with it: it reads governed entities from twelve modules and the catalog's documents, extracts a graph of entities, relationships, and community summaries, and serves that graph to its consumers.

flowchart LR
    subgraph SRC["Governed sources"]
        direction TB
        G["Glossary terms"]
        C["Catalog assets"]
        K["Data contracts"]
        P["Data products"]
        R["Processes"]
        F["Flows"]
        V["Events"]
        L["Policies"]
        D["Documents"]
        M["Metrics"]
        I["Reports"]
        SM["Semantic models"]
        BD["BI dashboards"]
    end
    SRC --> DNA(["Organisation DNA<br/>entities · relationships<br/>community summaries"])
    DNA --> H["HERC<br/>conversational AI"]
    DNA --> S["Semantic search"]
    DNA --> E["Graph exploration"]
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class DNA accent

Figure 1 — Twelve governed modules feed the graph; three consumers read it. DNA is the single place the AI's understanding of your organisation lives.

Why teams reach for this

When HERC answers a vague question like "how does customer churn propagate through our products?", DNA is what makes the answer specific to your organisation rather than generic. The richer and fresher the graph, the more precise the answer.

What DNA looks like¶

Surface	Where	What you see
Overview	`/dna`	Index status (running / done / failed), entity count, growth delta vs previous run, type distribution donut, source module coverage, run history, Re-index button.
Search	`/dna` → Search	Free-text input → ranked entity hits with scores.
Graph explorer	`/dna` → Graph	Interactive React Flow canvas: nodes coloured by type, edges show relationships, click-and-drag to pan, zoom to focus.
Entity detail	`/dna` → click entity	Name, type, description, embedding stats, neighbours list, source documents (term / asset / contract / …).

How DNA works¶

DNA is built on Microsoft GraphRAG. The pipeline runs in four stages, then loops incrementally as your data changes.

flowchart LR
    COL["Collect<br/>read governed entities<br/>chunk into a corpus"] --> IDX["Index<br/>GraphRAG · detached subprocess<br/>entities · relationships<br/>Leiden communities"]
    IDX --> ETL["ETL<br/>Parquet to PostgreSQL"]
    ETL --> SRV["Serve<br/>community summaries<br/>+ deep graph query"]
    SRV -. "weekly job · 50+ changed entities" .-> Q{"Re-index?"}
    Q -. "incremental (Update)" .-> COL
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class IDX accent

Figure 2 — The build/refresh lifecycle. The first run is a full build; every run after is incremental — only new or changed entities are processed and merged into the existing graph.

Collect. A background job reads the active, published governed entities from across the platform — published glossary terms, catalog assets, contracts, products and processes, plus active flows, confirmed events and active policies, and the processed documents of published catalog assets (of a versioned asset, only its active version). A document goes in as metadata only: its title, file name, asset, kind, language and the topics the AI analysis found. Its text, its summary and the names it mentions stay out, because anyone who may use the Brain can ask it about what it holds, without the catalog permission that document search requires. Published metrics, reports and semantic models that have an active version, and every active BI dashboard, go in the same way: by name and description, plus a report's tags. A report is read under its active version's name, never a draft's. A query result, a report's cards, a model's definition and a dashboard's link stay out. A BI dashboard is read whatever dashboard group it is in, and the Data sources card says so. Drafts, under-review, deprecated and archived rows are excluded. It chunks the text and adds it to a corpus, and records per source how many items it read out of how many there are (the Data sources card).
Index. GraphRAG runs in a detached subprocess (so it survives uvicorn reloads) and produces a graph: extracted entities, relationships between them, and Leiden community clusters with hierarchical summaries.
ETL. The Parquet outputs are loaded into PostgreSQL — entities, relationships, communities, text units — so the platform can query the graph at low latency.
Serve. HERC's specialist agents read community summaries on every prompt; the deep "search the graph" tool is available on demand.

Incremental by default

After the first run, subsequent runs are incremental — only new / changed entities are processed and merged into the existing graph. A weekly Procrastinate job (Sunday 02:00) checks the change threshold (50+ modified entities) and triggers a re-index automatically. It counts changes in every source the Brain reads (glossary terms, catalog assets, contracts, products, events, policies, flows, processes, documents, metrics, reports, semantic models and BI dashboards), and only in items the Brain would read: editing a draft does not count.

Setup — what an admin needs to do once¶

Prereq	Where	Why
AI provider key	AI & Agents → Provider Keys	GraphRAG calls the default model's provider to read your content and extract entities.
OpenAI key, for embeddings	AI & Agents → Provider Keys	The Brain indexes what it reads with OpenAI `text-embedding-3-small`, so the text of your content is sent to OpenAI. With an OpenAI default model one key does both; with Anthropic or Datahub Private AI (neither provides embeddings) add an OpenAI key as well.
Roles	`/rolegroups`	`dna.read` to view the dashboard / search / graph; `dna.write` to trigger a re-index.
Initial index	`/dna` → Re-index	Built automatically on first start, but admins can trigger one manually after a large initial import.
(Optional) Private AI	License	When Private AI is licensed and provisioned, the Brain reads your content with the platform model, whatever default model you chose for the agents. Embeddings still need your OpenAI key, because Private AI provides none, so the text of your content still goes to OpenAI. Once this month's Private AI usage reaches a hard monthly token limit, the Brain does not build until the limit is raised or the month ends. See AI platform.

Triggering a re-index¶

/dna → Re-index. The button is gated on dna.write and is disabled if a run is already pending or running, or if the Brain cannot build with the current AI setup: the header then names what is missing (see Troubleshooting), and the button's tooltip repeats it. Once your plan's build limit is reached, the page offers no Re-index until the next build may start (see How often the Brain can build). The job is deferred to Procrastinate, so it survives a uvicorn restart between the API ack and the indexing kickoff.

stateDiagram-v2
    direction LR
    [*] --> Idle
    Idle --> Pending: Re-index (manual or weekly)
    Pending --> Running: Worker picks up the job
    Running --> Done: Index + ETL succeed
    Running --> Failed: Error, crash, or OOM
    Failed --> Pending: Retry indexing
    Done --> Idle: Ready for the next run
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class Running accent

Figure 3 — An index run's status lifecycle. A run stuck in *Running* after a crash is cleaned up on startup and marked *Failed* with an actionable message so you can *Retry indexing*.

When to trigger manually:

After a bulk import of glossary terms or catalog assets.
After publishing a new contract or restructuring an existing one.
After a major taxonomy change (renamed domains, new tag scheme).
If HERC's answers feel stale — check the Last successful run timestamp.

A run typically takes 1–10 minutes depending on corpus size and provider latency. Progress streams to the dashboard.

How often the Brain can build¶

Your licence plan sets how many Brain builds may succeed in a rolling window:

Plan	Builds that may succeed
Standard, and a trial or a licence that names no plan	1 in any 7 days
Professional	3 in any 7 days
Enterprise	1 in any 24 hours

Every build counts the same way, whatever started it: the weekly schedule, new content, startup or Re-index by hand. Only a build that succeeds uses one. A failed build does not, nor does a build that found too little content to build; the build history says so on that build's row. The first build always runs.

While a build can start, the Re-index button's tooltip names the limit and says that a build that succeeds uses one. Once the limit is reached, the header says Build limit reached, with your plan's limit and when the next build can start, and the page offers no Re-index until then. The weekly and new-content builds skip until then too, and run on their next turn after it.

Which model built the Brain¶

The Model slot in the /dna header names the model that read your content for the graph on screen, and the embeddings model its text was sent to. Open it for the full sentence; when the AI setup has changed since, it also says what the next build will use. Each run records its models when GraphRAG starts, so the answer stays true after the configuration changes. A graph built before runs recorded their models shows Not recorded; a run that stopped before calling any model (too little content) records none.

What's in the dashboard¶

Card	What it tells you
Status	Idle / pending / running / failed. If failed, click for the actionable error message.
Entity count	Total extracted entities; growth delta vs previous run.
Type distribution	Donut: which entity types dominate (e.g. concept, organisation, system, role).
Data sources	Per source, what the last reindex read: 3 of 12 indexed, with the rule that decided it, such as Only published terms that are not deprecated are read or Only confirmed events are read. Sources that gave nothing are listed too, with the reason: nothing in them yet, or none of their items meet the rule. The card appears after the first reindex even when that reindex found nothing to read, which is when it matters most. Use it to see why a term or an event you expected is missing: publish or confirm it, then reindex.
Build history	The latest builds: what started each one (weekly schedule, new content, startup or by hand), where it ran (the Datahub app or a separate build process), when it started and finished, and its status. A failed build gives its error, and a build that found too little content says it did not count toward your plan's build limit. A build that stopped reporting progress shows as failed, with why, rather than as running.

The graph explorer¶

The graph view is a React Flow canvas:

Nodes are entities, coloured by type.
Edges are relationships extracted by GraphRAG (e.g. Customer → uses → Product).
Click a node to see its neighbours, type, description, and source.
Drag to pan, scroll to zoom, double-click an entity to focus on it.

It's the AI's view, not a schema

The graph is intentionally not a full UML diagram — it's the AI's understanding of your business, useful for sanity-checking that the right concepts are connected.

How DNA powers HERC¶

Every conversation HERC has uses DNA in two ways:

Lightweight context injectionOn-demand deep query

The community summaries (multi-level abstractions of your knowledge graph) are added to the agent prompt automatically on every conversation. This makes HERC's answers organisation-specific without you having to think about it.

When a user asks something specific ("how does churn relate to subscription tier?"), HERC can call the query_brain MCP tool. In its local mode it matches each word of the question on its own against the names and descriptions of the graph's entities, leaving out question words such as what, is or wat, and returns the best matches with their relationships: entities that match every word come before those that match some, and matches in the name come before matches only in the description.

Why DNA quality matters

DNA quality directly affects answer quality — a thin or stale brain produces thin answers. Keeping the index fresh (weekly incremental, or a manual re-index after a big change) is the single biggest lever on how specific HERC can be.

Limitations¶

Limit	Why	Workaround
Re-indexing has cost (provider calls).	Entity extraction + embeddings call the LLM.	The incremental update keeps cost down; full rebuilds are rare.
GraphRAG is one-graph-per-tenant.	Single workspace per tenant.	If you operate multiple business units, consider separate tenants.
Search matches words, not meaning.	Each word of the question is matched as written (a part of a word counts, so afval finds verpakkingsafval); a synonym or a translation does not match.	Name things the way your glossary does, or add the words people use to a glossary term's description so the index picks them up.
DNA doesn't index transcripts yet, and knows a document only by its metadata (title, file, asset, kind, language, topics).	Everyone who may use the Brain could otherwise ask it what a document says, without the catalog permission that document search requires.	For what a document says, use document search in the catalog, which checks that permission. Tag transcripts with the right domain.
DNA knows a metric, report, semantic model or BI dashboard only by its name and description (and a report's tags), and reads a BI dashboard whatever dashboard group it is in.	Everyone who may use the Brain could otherwise ask it about a query result, a report's cards or a model's definition, without that module's read role or the dashboard's group.	Keep anything confidential out of their names and descriptions. Open the item in its own module for its content: that module checks the read role and the dashboard group.
The graph rebuilds from scratch only when explicitly requested.	Stable IDs across runs.	Re-index manually after a major restructure.

Audit & compliance¶

Question a CISO might ask	Where to look
"Where does indexing run?"	A detached subprocess in the platform's compute. The provider is called for embeddings + extraction.
"Are credentials stored on disk?"	No. The runtime config uses `${...}` placeholders; real credentials are passed via subprocess env vars.
"Can a viewer see entities they don't have access to?"	The graph is platform-wide, but specialist agents filter results by the user's role. The graph explorer is gated on `dna.read`.
"Where is the index stored?"	PostgreSQL (`dna.brain_*` tables) + a workspace folder for GraphRAG's internal state.
"Did any data leave the tenant?"	Yes, to the providers named in the Model slot on `/dna`: the model that read the content, and OpenAI, which receives the text of your content to compute embeddings. With Private AI, reading stays on the platform, but embeddings still go to OpenAI.
"Which model built this graph?"	The Model slot on `/dna`, or `built_with` on `GET /dnamodule/brain/status` (per run: `GET /dnamodule/brain/runs`).
"What started this re-index?"	The Build history card on `/dna`: by hand, the weekly schedule, new content or startup, with when it started and where it ran (`GET /dnamodule/brain/runs`).

Troubleshooting¶

Symptom	Likely cause	Fix
Index status stuck on Running after a server restart	Stale run from a crashed previous process.	Cleanup runs automatically on start; if not, click Retry indexing.
Header says AI provider required	No provider key and no default model at all.	AI & Agents → Provider Keys: add a key.
Header says OpenAI key required	The default model is Anthropic or Datahub Private AI, which provide no embeddings, and there is no OpenAI key.	AI & Agents → Provider Keys: add an OpenAI key.
Header says Anthropic key required (or another provider)	The default model belongs to a provider with no active key.	Add that provider's key, or choose a model you have a key for under AI & Agents → Model Settings.
Header says Private AI unavailable	The default model is Datahub Private AI, which this environment does not offer.	Choose an OpenAI or Anthropic model under AI & Agents → Model Settings, or contact Datahub.
Header says Brain not on Azure OpenAI yet	The default model runs on Azure OpenAI, which the Brain cannot build with yet. HERC and the agents still run on your Azure deployment. See AI platform.	To build the Brain now, choose an OpenAI or Anthropic model under AI & Agents → Model Settings.
Header says AI model required	Private AI or a local AI server is set up with no model.	Choose a default model under AI & Agents → Model Settings.
Header says Private AI limit reached	Private AI reads your content, and this month's Private AI usage has reached the hard monthly token limit. The agents are refused on the same limit. The Brain does not switch to another model instead.	Raise the monthly token limit under AI & Agents → Provider Keys, or build again from the 1st of next month.
Header says Private AI trial limit reached	As above, but the limit is the Private AI trial's, which cannot be raised during the trial.	Contact Datahub about a paid plan.
Header says Build limit reached	Your licence plan's builds for this window are used up (see How often the Brain can build).	Wait until the time the header names; the Brain then builds on its next scheduled turn, or Re-index by hand. For more builds, contact Datahub about a higher plan.
Re-index rejected up-front (HTTP 422)	Same as the header message; the API refuses to enqueue a run it cannot build. `detail.code` names the reason (`no_provider`, `no_embedding_key`, `no_completion_key`, `private_ai_unavailable`, `azure_openai_unsupported`, `private_ai_quota_reached`, `private_ai_trial_quota_reached`, `no_model`, `index_limit_reached`); for `index_limit_reached`, `detail.index_limit` says when the next build may start.	Fix the reason, then re-index; for the build limit, wait until that time.
Search returns nothing for a paraphrase	Search matches the words of the question, not their meaning.	Use the words the entity's name or description uses, or add the literal terms to a glossary term description so the index picks them up.
Entity count flat after re-index	No qualifying changes since the last run.	Check the run history: did anything new land?
Re-index button greyed out	A run is pending or running, or the Brain cannot build (the header says why).	Wait for the run to finish, or fix what the header names. Without an AI role, ask an AI administrator.
Graph explorer is empty	Brain hasn't been indexed yet.	Trigger a re-index.
"Server restarted during indexing" error message	Crash or OOM during a previous run.	Click Retry indexing. The startup cleanup writes this message; it's safe to re-run.