Metadata Engine: connections, snapshots, tests, freshness¶

Connect a data source and Datahub continuously discovers its schemas, snapshots them, runs tests, profiling, and freshness checks, and promotes what matters into the Catalog — so stewards always see what's there now and what changed.

If the Catalog is what we know about our data, the Metadata Engine is how we know it — the bridge from your physical data sources to governed catalog assets.

ConnectorsADLS · PostgreSQL · Databricks · dbt

SnapshotsPoint-in-time · diffable

Quality checksTests · Profiling · Freshness

SchedulerCron · 1-min tick

When to choose this¶

Connect a data source

Register an ADLS Gen2, PostgreSQL, Databricks Unity Catalog, or dbt-artifacts connection — all first-class types in this module. Secrets live in Azure Key Vault, never in our database, and most ADLS / dbt connections share a single service principal.
Discover a schema

Extract the source's schemas, tables, and columns, snapshot the result, and promote what you want into the Catalog in one flow.
Detect schema drift

Compare today's snapshot to last week's; the diff shows added / removed / renamed columns and type changes.
Run data tests

Uniqueness, not-null, accepted values, custom SQL — schedule them, and alert on failure.
Monitor freshness

"The orders table should have new rows in the last 30 minutes." If it doesn't, raise an alert.
Profile for anomalies

Row count, null %, distinct count, min / max / mean — tracked across snapshots with anomaly detection.

When you don't need the Metadata Engine

Manually-registered catalog assets → you can register in the Catalog without any connection.
Querying data → the engine reads metadata, not row contents.
BI tools or dashboards → use the Dashboards module instead.

How the engine works¶

A connection feeds a repeating loop. The engine extracts the live schema, snapshots it, compares it for drift, runs the quality checks you've configured, and promotes the pieces that matter into the Catalog. The scheduler drives the loop so it keeps happening on its own.

flowchart LR
    CONN["Connection<br/>ADLS · PostgreSQL"] --> EXT["Extraction<br/>schemas · tables · columns"]
    EXT --> SNAP["Snapshot<br/>point-in-time metadata"]
    SNAP --> CMP["Compare<br/>drift: added · removed · renamed"]
    SNAP --> QA{{"Quality checks<br/>Tests · Profiling · Freshness"}}
    SNAP --> PROMO["Promote<br/>selected schemas / tables"]
    PROMO --> CAT[("Data Catalog<br/>draft → approved assets")]
    SCHED(["Scheduler<br/>cron per connection"]) -.drives.-> EXT
    SCHED -.drives.-> QA
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class SNAP accent

Figure 1 — One ADLS or PostgreSQL connection drives a loop: extract the live schema, snapshot it, compare for drift, run quality checks, and promote what matters into the governed Catalog. The scheduler repeats it per connection. Databricks and dbt connections follow a sync model instead — each sync is a versioned run rather than a snapshot/diff (see Connection types below).

Why teams reach for this

You get continuous discovery, schema-drift detection, and data-quality checks in one place — feeding straight into a governed Catalog. Every snapshot is a point-in-time record you can diff against any other, so "what changed, and when?" is always answerable.

Connection types¶

Four source types are first-class in this module. Two follow the snapshot loop in Figure 1; two follow a sync model.

Type	What it connects to	Model	Setup guide
ADLS Gen2	Delta tables in Azure Data Lake Storage Gen2, read directly via `deltalake`.	Extract → snapshot → diff loop	This page
PostgreSQL	A Postgres database over a live connection.	Extract → snapshot → diff loop	This page
Databricks Unity Catalog	UC assets and columns, table / column / external lineage, and query usage — read from Databricks system tables.	Versioned sync runs (`all` / `schema` / `lineage` / `query_usage`)	Databricks OAuth
dbt artifacts	Pre-generated `manifest.json` / `catalog.json` / `run_results.json` — every model, source, seed, snapshot, plus lineage and test results. No dbt run inside Datahub.	On-demand or scheduled artifact sync runs	dbt Integration, dbt on Azure Blob

All four appear in the same Connections grid; opening a Databricks or dbt connection lands on its own detail page (/metadata-engine/databricks-connections/{id}, /metadata-engine/dbt-connections/{id}).

Microsoft Fabric and Azure SQL: harvested from the integration¶

A Microsoft Fabric warehouse or an Azure SQL (or SQL Server) database is not set up here: its host, database and credentials stay on its integration (Admin → Integrations). A Metadata Engine administrator opens that integration and clicks Harvest. Datahub then reads the database's tables, views, columns and keys: primary, unique and foreign keys, including Fabric's NOT ENFORCED keys. Only the catalog is read, never rows.

The first harvest adds a connection of type Azure SQL or Fabric to the Connections grid. It holds no credentials. It shows the harvested catalog and snapshots, but it cannot Test, Extract or be edited, because the integration owns the connection. The harvest card on the integration page shows the last harvest, what it stored, and keys it could not read. A semantic model on that integration checks its tables and keys against the harvest.

Shared service principal — configure once

ADLS and dbt-on-Azure connections authenticate with one shared service principal (an Azure AD app registration: tenant ID, client ID, client secret). You set it up once under /metadata-engine/setup → Authentication; its secret lives in Key Vault under a single fixed key, and individual connections then need no secret of their own — you just give a storage account and pick a container. Per-connection ("custom") credentials remain supported as a fallback, but the two are never mixed: a connection uses either its own credentials or the shared one, never a silent fallback.

Databricks is different — it stores no credentials at all. A Databricks connection references a Databricks workspace you register once (it supplies the host, SQL warehouse, and auth); the connection itself only records the workspace, the Unity Catalog name, and an optional schema filter.

What the Metadata Engine looks like¶

Surface	Where	What you see
Overview	`/metadata-engine`	KPIs (connections, snapshots, recent failures), trend chart of snapshots over the last 14 days, top-status connection cards.
Setup	`/metadata-engine/setup`	Guided setup: Authentication (shared service principal), Private Endpoints, ADLS, dbt, and PostgreSQL connections.
Connections	`/metadata-engine/connections`	Grid of connection cards across all four types — name, type, last test status, last snapshot, schedules.
Connection detail	`/metadata-engine/connections/{id}`	ADLS / PostgreSQL tabs: Overview, Extraction, Snapshots, Catalog, Schedules, Tests, Profiling, Freshness, Tasks.
Databricks connection	`/metadata-engine/databricks-connections/{id}`	Synced UC assets; per-asset Schema, Lineage, Sample, and Query Usage tabs.
dbt connection	`/metadata-engine/dbt-connections/{id}`	Synced dbt nodes; Schema, Lineage, dbt (SQL viewer), and Data Quality tabs.
Governance coverage	`/metadata-engine/governance`	Per-connection and total documentation / quality coverage with dated trend charts.
Snapshots	`/metadata-engine/snapshots`	List of all snapshots across connections; filter by connection, compare any two.
Snapshot detail	`/metadata-engine/snapshots/{id}`	Full schema → table → column hierarchy with stats.
Data tests	`/metadata-engine/data-tests`	All test definitions across connections; filter by status, latest run results.
Insights (profiling)	`/metadata-engine/insights`	Profiling history and trend charts for any catalog table.
Freshness	`/metadata-engine/freshness`	Per-table freshness rules and recent check results.
Scheduler	`/metadata-engine/scheduler`	All schedules across connections; recent runs.
Task logs	`/metadata-engine/tasks`	Background task history (extraction, snapshot, test, profiling, freshness runs). A task that is still pending says why it has not started: how many metadata jobs start before it, that it waits for a running job or for its connection's earlier jobs, or that it was never queued.

Concepts¶

Concept	What it is
Connection	A registered data source — ADLS Gen2, PostgreSQL, Databricks Unity Catalog, or dbt artifacts — with type-specific config. Secrets, where a connection needs its own, live in Key Vault.
Adapter	The connector for a connection type. ADLS is read directly as Delta (via `deltalake`); PostgreSQL over a live connection; Databricks reads Unity Catalog system tables; dbt reads pre-generated artifact files.
Shared service principal	One Azure AD app registration (tenant ID, client ID, client secret) configured once and reused by every ADLS and dbt-on-Azure connection. Secret stored in Key Vault under a single fixed key; per-connection credentials remain supported as a fallback.
Session	A working dbt project Datahub prepares per connection for tests, profiling, and freshness. It's created automatically at connection time and refreshed / rebuilt on demand — there is no manual setup step.
Extraction	A live discovery of the source's schemas / tables / columns. Produces a working snapshot you can review before saving.
Snapshot	A persisted point-in-time capture of metadata, linked to a connection. Used for diffs and history.
Catalog (per-connection)	A deduplicated view of every schema / table / column ever seen on the connection — soft-deletes columns when they disappear, and retains their history.
Data test	A test definition (uniqueness, not-null, accepted values, custom SQL) on a column or table.
Profiling run	A row-count plus per-column null / distinct / min / max / mean capture, used for anomaly detection.
Freshness rule	"Column X must have a value within the last N minutes / hours / days."
Schedule	A cron expression plus which tasks (extract, test, profile, freshness) to run, per connection.
JobRun	A single execution of a schedule, with start time, duration, status, per-task results, and any error.

Setup — what an admin does once¶

Prereq	Where	Why
Azure Key Vault	Configured by the platform — credentials live there.	Connection secrets never touch the database.
Shared service principal (for ADLS / dbt-on-Azure)	`/metadata-engine/setup` → Authentication	Configure one Azure AD app registration once; ADLS and dbt connections reuse it instead of each carrying a secret.
At least one connection	`/metadata-engine/connections` → New connection	The engine is empty until you connect a source.
Roles	`/rolegroups`	`metadata_engine.resources.read`, `metadata_engine.resources.write`. Both default to Administrator. Read opens `/metadata-engine/setup` and shows the connections and their sync status. Adding, editing, testing, syncing, archiving and deleting a connection take write: a reader is not offered them, and the server refuses them.
Scheduler enabled (automatic)	The platform runs a periodic tick that picks up due schedules.	No admin action required.

Creating a connection¶

ADLS Gen2PostgreSQLDatabricks UCdbt artifacts

Configure the shared service principal once under /metadata-engine/setup → Authentication (or supply per-connection credentials instead).
/metadata-engine/connections → New connection → ADLS. Provide a name and storage account; leave tenant / client / secret empty to use the shared service principal, or fill them for custom credentials.
Discover containers — Datahub lists the containers reachable with those credentials.
Test connection — Datahub calls Azure to confirm access. If it works, you'll see a green check.
Save.

Register the database once as a PostgreSQL integration (Admin → Integrations): host, port, database, login, password and SSL mode. Publish it and test it there.
/metadata-engine/setup → PostgreSQL → Add connection. Pick the integration and, optionally, a display name. No host or password is entered here: the connection reads them from the integration every time it connects, so a password goes only to the host it was entered for.
Create — Datahub issues SELECT 1 through the integration before it saves the connection.

An integration that already has an engine connection, is not published, or is restricted to role groups is shown but cannot be picked. To change the host or the login, edit the integration. A connection created before this change that matched no integration keeps its own host and login; changing its host asks for the password again. If you delete the integration a connection reads from, that connection can no longer connect: delete it, then create a new one from a PostgreSQL integration.

Register your Databricks workspace once (it supplies host, SQL warehouse, and auth — see Databricks OAuth).
/metadata-engine/setup → Databricks → Add connection. Pick the workspace, then pick one or more Unity Catalog catalogs from the workspace's list, or All catalogs. The list shows the catalogs that the user or service principal the workspace's integration signs in as can see. All catalogs syncs every catalog in that list, listed again at each sync, so a catalog added later is synced too. It skips Databricks' own system, samples, hive_metastore and __databricks_internal catalogs, which can still be picked by name. With exactly one catalog picked you can enter a default schema filter; with several or all catalogs the sync reads every schema. Choosing another workspace clears the catalogs. If the list is empty or cannot be read, the field says why and lets you type the names; Type the names instead also works while the list is loading. A scoped access token needs the unity-catalog scope for the list. Without it, the field names the missing scope and takes typed names. The sync still reads tables, columns and lineage through the SQL warehouse, though external lineage may be missing until the integration's token has the scope. A saved catalog that is not in the list stays selected, and the field warns that a sync cannot read it. When the list has a catalog the name was most likely meant to be, such as sales_prod for sales-prod, the field names it and offers Use sales_prod. No secret is stored on the connection. The prerequisites and the Unity Catalog GRANT statements are in the Setup guide next to Add connection.
Connect & Test opens the new connection's page. Run Sync All there, or Sync on the connection's card in the list, without opening it. Datahub pulls UC assets, columns, lineage, and query usage into the catalog. A catalog name that needs back quotes in SQL, such as dbw-internal-001, syncs like any other. Databricks records lineage every time a table is read or written. The sync stores each lineage link once, with the most recent time it was used, which is also the last activity the connection page shows. The sync log shows table lineage and column lineage as separate steps. A connection with several catalogs syncs them one after the other. A catalog that cannot be read, for example because a grant is missing, does not stop the others: its tables stay in the data catalog, and the connection page names it under Catalogs not read. A catalog taken out of the connection is marked removed at its next sync.

While a sync runs, its connection card shows Syncing… and the notification bell lists it, also after you leave or reload the page. When it ends, the bell row shows the outcome; a failed sync's message names the sync, says why it failed and links to the task. When a Databricks sync stops on an error Datahub did not expect, the message says only that the sync step failed and that the task log has the details, because the raw error text can contain hosts or queries; the log keeps it. A short network drop or server error does not end it. If the server has not answered for 5 minutes, no longer knows the task, or you may not open it, the bell row and a message say the outcome is unknown. Signed out, nothing is checked; a sync still running when you sign back in is followed again.

On the Databricks tab of /metadata-engine/setup, a sync you start with Sync, or one already running for the connection, also shows under its card as on the connection's page. It shows the sync's state, why a pending sync waits, its progress, and how it ended. View opens the task. A user who may change metadata engine settings can Cancel it while it waits. A cancel that is refused says why, for example that the sync has already started.

Point Datahub at where your pipeline writes manifest.json / catalog.json / run_results.json (Azure Blob, HTTP, or local). Full walkthrough: dbt on Azure Blob.
New connection → dbt. Azure Blob sources reuse the shared service principal; HTTP sources can use a bearer token (stored in Key Vault).
Save, then Sync — Datahub reads the artifacts and surfaces every dbt node, its lineage, and test results. See dbt Integration.

Setup is transactional: if a Key Vault write fails, the database row is rolled back, so you never end up with an orphan connection.

Discover, snapshot, and promote¶

The standard "what's in this source?" loop, for ADLS and PostgreSQL connections (Databricks and dbt use Sync instead):

Open the connection → Extraction tab. There's no manual setup session step any more — Datahub prepares a working session automatically when the connection is created and refreshes or rebuilds it on demand before each run. (ADLS metadata is read directly as Delta; a lightweight dbt session backs tests, profiling, and freshness for both ADLS and PostgreSQL.)
Extract — the engine reads schemas / tables / columns from the source.
Save snapshot — persists the result. Each snapshot has a name (auto-generated or custom).
Browse — the Catalog tab shows the deduplicated inventory; expand any schema → table → columns.
Compare — Snapshots tab → Compare → pick two snapshots → see added / removed / renamed columns and type changes.
Promote to Catalog — select schemas / tables → Promote → catalog assets are created (in draft) with column metadata pre-filled. From there the standard catalog approval flow applies (see Data Catalog).

How promotion stays governed

Promotion never writes directly to the published Catalog. It creates draft assets with the discovered column metadata pre-filled; a reviewer must approve them before they're published. Connection ≠ asset — promotion is the explicit governance step, and ownership / approval are configured downstream in the Catalog.

Sessions are automatic

Older builds asked you to click Setup session before extracting or testing. That step is gone. A session is created in the background when you save a connection, and every extraction, test, profiling, and freshness run resolves (and if needed rebuilds) its own session — so all actions work directly, and a broken session self-heals on the next run.

Data tests¶

Data tests live per connection, in two flavours:

Built-in testsCustom SQL

Uniqueness, not-null, accepted-values, relationships. Pick a table and column; the engine generates the SQL for you.

Write the test as a SELECT that returns failing rows. An empty result set = pass.

Run them ad-hoc (Run now) or on a schedule. Failed tests show in the Tests tab and can trigger an alert rule — configure the rule with trigger source = data test.

A table's keys are tested too, as the table has them:

Key	What is tested
Primary key	The key the source defines. A primary key you declared is tested only when the source defines none; once the source's key takes its place, yours is kept but no longer tested. In the column table, PK marks the key that is tested.
Unique keys	The source's and the ones you declared. A set of columns named by both is tested once.
A key naming a column the table no longer has	Not tested at all, rather than on the columns that remain.

The primary-key and unique-key editors show this answer on every key: Tested by dbt or Not tested by dbt, followed by the reason. The reasons are: - the source's primary key is tested instead; - the source's primary key takes its place, but a column of that key is no longer in the source, so neither key is tested; - a column of the key is no longer in the source; - another unique key names the same columns; - the table is no longer in the source.

The PK badge in the Data Tests tab reads the same answer. Keys on a Databricks table show neither badge (see Limitations).

A run's tests can change without an edit

Because the tested keys follow the source, a re-sync can change which tests a run makes, and a run can report a failure it did not report before. A table whose only key is its primary key is included when all tables are tested and in scheduled runs.

Profiling¶

For every catalog table on a connection you can run profiling:

Row count plus per-column null %, distinct count, and min / max / mean (numeric).
Every run is retained, so the trend view can show deltas across runs with anomaly detection:

Anomaly	Severity	Trigger
Null % increased > 10pp	warning	A column got noticeably nullier.
Distinct count dropped to 0	critical	A column lost all its values.
Row count dropped > 50%	critical	Table shrank.

What the engine actually reads

Extraction reads only metadata — schema, columns, and types. Profiling reads aggregate statistics — row counts and per-column null / distinct / min / max / mean — never full row contents. A custom SQL test reads exactly what its query selects, and nothing more.

Freshness¶

Per-table freshness rules:

Pick a column (typically a timestamp like updated_at).
Set a threshold (e.g. 30 minutes).
The engine checks the newest value of that column on the cadence you schedule.
If the value is older than the threshold, the rule fails — visible on the Freshness tab and (optionally) wired to an alert rule.

Multiple rules per table are supported — use them when you have several timestamp columns that should each stay fresh.

Scheduler¶

Per connection you can schedule any combination of: extract → snapshot → tests → profiling → freshness. A schedule has a cron expression, a timezone, and a list of tasks. The scheduler tick (every minute) finds due schedules, defers them to the background task queue, and recomputes the next run time.

flowchart TB
    TICK["Scheduler tick<br/>every minute"] --> DUE{"Any schedule due?"}
    DUE -->|no| NEXT["Recompute next run"]
    DUE -->|yes| JOB(["JobRun starts"])
    JOB --> TASKS["Run configured tasks in order<br/>extract → snapshot → tests → profiling → freshness"]
    TASKS --> OK{"All tasks succeeded?"}
    OK -->|yes| DONE["JobRun completed<br/>per-task results + duration"]
    OK -->|no| FAIL["JobRun failed<br/>error message recorded"]
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class JOB accent

Figure 2 — A scheduled run. Every minute the scheduler checks for due schedules; a due one opens a JobRun that runs the configured tasks in order and records the outcome — completed with per-task results and duration, or failed with the error — for audit.

Each JobRun captures per-schedule history: start time, duration, status, the per-task results, and any error message.

Cancelling a scheduled run¶

Cancel a running JobRun from the Operations Console. The run is recorded cancelled ("Cancelled by operator"), and no later task of the run starts. The task it is on stops as follows:

Task	When it stops
Extract	Before its next schema or table. No snapshot is saved.
Data tests	Before their next dbt command. A dbt test command already running finishes, but its results are not recorded.
Databricks sync	At its next progress update.
Profiling, freshness, dbt sync	They finish first.

Cancelling an extract or data-test step from its own card stops it the same way, and cancels its run with it: the run is recorded cancelled with the reason, for example "Stopped during 'extraction': the step was cancelled", and no later task starts. A Databricks or dbt sync that is a step of a scheduled run can't be cancelled on its own, from its card: cancel the run instead.

Two rows for one run¶

While a schedule runs, the Operations Console lists it twice: a Metadata sync row for the run, named after the schedule, and an Internal job row named metadata_engine.execute_schedule for the background job that carries it out. Both rows have a Cancel, and they stop the run in different ways:

Cancel on	What happens	Use it
The Metadata sync row	The run's current task stops at its next check, as in the table above. Profiling, freshness and dbt sync finish first.	Normally.
The Internal job row	The run stops within seconds, whatever task it is on. What that task had not saved yet is dropped, and the run is recorded cancelled ("Cancelled by operator").	When the run must stop now: for example, it is in a long profiling task and a cancel on its Metadata sync row would wait for that task to finish.

The run doesn't stay running when its job stops another way either: a worker that shuts down before the run ends records it failed. A run whose tasks had all ended before the job stopped keeps its own result.

Deleting a schedule or connection mid-run¶

Deleting a schedule, or the connection it runs on, deletes the schedule's runs with it, including a run that is still going. That run does not fail. It stops the way a cancel stops it: an extract, data-test or Databricks task stops at its next check, any other task finishes first, and no later task starts.

flowchart LR
    DEL(["Schedule or connection deleted<br/>while a JobRun is running"]) --> TASK{"The current task<br/>stops at its next check?"}
    TASK -->|"yes: extract, data tests,<br/>Databricks sync"| NOW["JobRun cancelled during the task<br/>with the reason"]
    TASK -->|no| FIN["The current task finishes"]
    FIN --> LEFT{"Tasks left?"}
    LEFT -->|no| DONE["JobRun completed or failed<br/>as usual"]
    LEFT -->|yes| STOP["JobRun cancelled before the next task<br/>with the reason"]
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class NOW,STOP accent

Figure 3 — A run in flight when its schedule or connection is deleted. An extract, data-test or Databricks task stops at its next check, and the run is recorded cancelled, with the reason. Any other task finishes; with no tasks left the run is recorded as usual, otherwise it stops before the next task and is recorded cancelled. Either way it is deleted along with its schedule.

The reason names what was deleted, for example Stopped before 'profiling': its connection was deleted or Stopped during 'extraction': its schedule was deleted. It says this run was deleted when the run itself was. If an operator had already cancelled the run, that cancel stands.

A run started at the moment of the delete

A run that started at the very moment of the delete can still show as running in the Operations Console. It stops at its next check by itself. To stop it sooner, Cancel it there like any other run.

Limitations¶

Limit	Why	Workaround
Connectors are ADLS Gen2, PostgreSQL, Databricks Unity Catalog, and dbt artifacts. Fabric and Azure SQL are harvested through their integration.	Other sources aren't natively connected yet.	Materialise other sources to one of these, or surface them via dbt artifacts.
Fabric and Azure SQL are harvested only when someone clicks Harvest.	No schedule runs these harvests yet.	Harvest again after the database's tables or keys change.
On Fabric, Azure SQL and PostgreSQL, a table name must match the harvested spelling exactly, case included.	The engine does not store the database's collation.	Use the spelling the harvest found; a semantic model's save names it.
Snapshot / drift diffing applies to ADLS and PostgreSQL only.	Databricks and dbt connections track versioned sync runs, not point-in-time snapshots.	Use the sync-run history (and Governance coverage trends) to see how a Databricks / dbt source changed over time.
Credentials are write-once via the UI.	Key Vault rotation is not automated through the engine.	Rotate the secret in Key Vault directly; the engine reads on next connection.
Snapshots store metadata only — no row contents.	Privacy + storage.	Use Insights / a profiling run for sample stats; row data stays in the source.
Profiling samples up to a configurable cap per column.	Avoids running out of memory on huge tables.	Pre-aggregate in the source if you need full-population stats.
A Databricks sync keeps at most the 20,000 most recently used lineage links per schema, for tables and for columns separately.	Databricks keeps a year of lineage, and a schema that is rebuilt every day, or read by a dashboard all day, can have far more links than one sync can hold in memory.	The sync log says when a schema reached the limit. The links left out are the ones used least recently.
A Databricks connection's default schema filter applies only when it syncs exactly one catalog.	A schema name means one schema in one catalog.	Pick one catalog to filter to a schema, or add a connection per catalog that needs a filter.
Keys of Databricks tables are not tested.	Data tests run through dbt, and the engine has no dbt adapter for Databricks yet.	Check the key in your own Databricks pipeline, or with an alert on a query that returns duplicate keys.
Schedules can't run sub-minute.	Cron plus the one-minute tick interval.	Use Alerts for sub-minute alerts.
Each connection requires its own owner / approver setup downstream in the Catalog.	Connection ≠ asset; promotion is the explicit governance step.	Promote in batches once the snapshot looks right.

Audit & compliance¶

Question a CISO might ask	Where to look
"Where do connection secrets live?"	Azure Key Vault, never in our database.
"Did the engine read sensitive rows from production?"	No — extraction reads only metadata (schema, columns, types); profiling reads row counts and column statistics only; tests read what your test SQL specifies.
"When did the customers table last change?"	Catalog tab → table → Column history; or Snapshots → compare two snapshots.
"Who set up this connection?"	Connection detail → audit metadata.
"Show every freshness check that failed last month."	Freshness tab → filter by status + date range.
"Can someone bypass approval and write directly to the Catalog?"	No — promotion creates draft assets; publishing requires reviewer approval.

Troubleshooting¶

Symptom	Likely cause	Fix
Test connection fails on ADLS	Wrong tenant / client ID / client secret, or RBAC on the storage account is missing.	Verify the App Registration credentials and the Storage Blob Data Reader role on the account.
Every scheduled dbt / ADLS run suddenly fails, and Authentication shows Credentials rejected	The shared Service Principal's client secret expired or was revoked in Entra ID.	A Rotate credentials task is already in the admin's inbox. Create a new secret on the App Registration, paste it under Setup → Authentication, save, then Test connection — that clears the status and closes the task.
Extraction returns 0 schemas on PostgreSQL	The user lacks `USAGE` on the schemas.	Grant `USAGE` on the schemas + `SELECT` on the tables to the connection user.
A PostgreSQL extraction ends with "Extraction complete; keys not readable, the stored keys were kept"	The tables and columns were read, but the query for their primary, unique and foreign keys failed. The keys stored before were kept, not dropped.	Check that the connection user can read `pg_catalog.pg_constraint`, then run the extraction again.
Snapshot diff shows wrongly-renamed columns	Heuristic match.	Ignore — promote columns explicitly to override.
Freshness rule failing all the time	The column the rule watches isn't actually a freshness signal (e.g. it's nullable).	Pick a different column or change the threshold.
Profiling run errors with a `Decimal` cast	Numeric column has unusual precision.	Expected behaviour after the platform's coercion patch — refresh; if it persists, file a ticket.
Schedule shows running but never completes	The background worker died mid-run.	Cancel the JobRun, restart the worker, and re-run the schedule.
A task stays pending and says "Queued: 3 metadata jobs start before it" or "Next in line"	Metadata jobs run a few at a time, and the ones before it are still running. A connection also runs one job at a time, so a task can wait for its own connection's earlier jobs.	Nothing to do: it starts by itself when its turn comes. A job you start yourself goes before queued scheduled runs.
A pending task says "It was never queued, so it will not start."	Queuing it failed when it was created, so nothing will ever run it.	Cancel it on its card and run it again.
A pending task says "Its queue job ended without running it, so it will not start."	Its job stopped before the task began, for example when the worker was shut down at that moment.	Cancel it on its card and run it again. A job that failed or was aborted also gets its task marked failed within a couple of minutes.
A pending task keeps saying "Starting now."	A worker took its job and then stopped, so the task never began.	Nothing to do: Datahub notices the stopped worker and marks the task failed within about 7 minutes. Then run it again.
A run of a deleted schedule or connection still shows running in the Operations Console	It started at the very moment of the delete and is finishing its current task.	Nothing to do: it stops before its next task and is recorded cancelled. To stop it sooner, Cancel it there.
A Databricks sync's message or bell row says "outcome unknown"	Datahub stopped following the sync: the server did not answer for 5 minutes, no longer knows the task, or you may not open it. The sync itself may have finished.	Open the task from the message (or Metadata Engine → Tasks) to see how it ended. If the task is gone, run the sync again.
A Databricks sync fails: "…still starting, or too busy to run it yet"	The SQL warehouse never picked up the query: it is cold-starting (which can take several minutes) or saturated.	Wait until the warehouse shows Running, then run the sync again.
A Databricks sync fails: "A harvest query was still running after … the query is slow on it"	The warehouse is up and ran the query, but it did not finish in time. Retrying on the same warehouse usually times out again.	Run it when the warehouse is less busy, give the connection a larger warehouse, or sync fewer schemas at once.
A Databricks sync fails: "The server running this task stopped before it finished. Run it again."	The server restarted while the sync ran, for example during an update or after running out of memory. Nothing from that sync was saved.	Run the sync again. If it stops at the same step again, run Sync Schema on the connection page instead, which skips lineage, and contact support with the task link.
A Fabric or Azure SQL harvest fails: "The database returned no tables …"	The integration's login sees no tables. Nothing was saved, and the tables stored before were kept.	Grant the login (or Fabric's service principal) read access to the tables, then harvest again.
A harvest stored the tables, but says the keys could not be read	The login may read the tables but not the key catalog (`sys.foreign_keys`, `INFORMATION_SCHEMA` constraints). The keys stored before are kept.	Grant `VIEW DEFINITION` on the database to the login, then harvest again.
The harvest card says the database declares no keys	The harvest ran, and the login sees no primary, unique or foreign keys.	Declare the keys in the database, or declare them in the Metadata Engine.