dbt Integration: nodes, lineage, tests, SQL¶

Read your dbt project's artifact files and make every model, source, seed, and snapshot a first-class object in Datahub — browse nodes, view raw and compiled SQL, trace lineage, and inspect test results, all without running dbt inside Datahub.

If your warehouse holds the transformed data and your dbt project describes how it was built, the dbt integration is what makes that description visible to the rest of your organisation.

Node typesModel · Source · Seed · Snapshot

Artifactsmanifest.json · catalog · run_results

SyncOn-demand · versioned per run

AccessRead-only explorer

When to choose this¶

Browse every dbt node

Explore all models, sources, seeds, and snapshots with their schema, materialization, tags, FQN path, and column metadata.
Read raw + compiled SQL

See the raw SQL from schema.yml and the compiled SQL after Jinja rendering, per node.
Trace lineage

Visualise which models depend on which sources or models, and which downstream models a source feeds.
Monitor test results

See whether each dbt test passed, failed, warned, or errored — one snapshot per sync run.

When not to reach for the dbt integration

Running dbt — Datahub does not execute dbt commands.
Managing dbt profiles or projects — that stays in your own infrastructure.
Scheduling dbt jobs — trigger syncs from your own CI/CD pipeline.
Importing dbt metadata into the Data Catalog — promotion from dbt to the Catalog is not yet wired; the dbt integration is a read-only explorer.

How it works¶

Your dbt run emits artifact files; you upload them to Azure Blob Storage; Datahub downloads, parses, and persists them as a versioned Sync Run you can browse.

flowchart LR
    M["manifest.json<br/>required — nodes · SQL · parent_map"] --> UP
    C["catalog.json<br/>optional — column types"] --> UP
    R["run_results.json<br/>optional — test outcomes"] --> UP
    UP["Upload to<br/>Azure Blob"] --> SYNC(["Datahub sync<br/>download · parse · persist"])
    SYNC --> RUN[("Sync Run<br/>versioned snapshot")]
    RUN --> BROWSE["Browse in Datahub<br/>nodes · lineage · SQL · tests"]
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class SYNC accent

Figure 1 — Your dbt pipeline emits artifacts; you (or your CI/CD) upload them to Azure Blob; a sync downloads and parses them into a versioned Sync Run whose nodes, lineage, SQL, and tests become browsable in Datahub.

Your dbt project generates artifact files (manifest.json, catalog.json, run_results.json) during a dbt run / dbt test / dbt docs generate cycle.
You — or your CI/CD pipeline — upload those files to an Azure Blob Storage container.
You trigger a sync in Datahub; the platform downloads the artifacts, parses them, and persists the results as a Sync Run.

Every sync is a versioned snapshot

Each sync run captures the full state of your nodes, columns, lineage, and test results at that moment. Select a previous run from the version picker to compare nodes and test results across runs — historical syncs are preserved, never overwritten.

What Datahub integrates from dbt¶

Each node becomes a first-class object with four detail tabs, each backed by a specific artifact.

flowchart TD
    NODE(["dbt Node<br/>identified by unique_id"])
    NODE --> SCHEMA["Schema tab<br/>columns · types · descriptions"]
    NODE --> SQLTAB["dbt tab<br/>raw + compiled SQL"]
    NODE --> LIN["Lineage tab<br/>upstream / downstream"]
    NODE --> DQ["Data Quality tab<br/>test pass / fail / warn / error"]
    classDef accent fill:#ff8400,stroke:#e67700,color:#ffffff,font-weight:600;
    class NODE accent

Figure 2 — What one synced dbt node yields: schema and SQL from the manifest, database-actual column types from the catalog, lineage from parent_map, and test outcomes from run_results.

Category	What you get
Nodes	All models, sources, seeds, and snapshots — name, schema, database, resource type, materialization strategy, tags, and FQN path.
SQL	Raw SQL (from `schema.yml`) and compiled SQL (from `dbt compile` / `dbt docs generate`) for each node.
Columns	Column names, data types, and descriptions. Manifest provides descriptions; `catalog.json` enriches the types with database-actual values.
Lineage	Directed dependency graph: upstream and downstream edges derived from `parent_map` in `manifest.json`.
Test results	Pass / fail / warn / error status per dbt test, with failure count, message, and execution time — one snapshot per sync run.

What the dbt integration looks like¶

Surface	Route	What you see
dbt Connections	`/metadata-engine/dbt-connections`	Grid of configured dbt artifact connections with name, last sync timestamp, and node count.
Connection detail	`/metadata-engine/dbt-connections/{id}`	Node browser, lineage tab, SQL viewer, test results, and sync history for one connection.
Node list	Node tab on the connection detail	Filterable, searchable table of all dbt nodes for the selected sync run.
Node detail	Click a node	Schema tab (columns + types + descriptions), dbt tab (raw + compiled SQL), Lineage tab (dependency graph), Data Quality tab (test results for this node).
Sync history	Sync Runs section	List of past syncs with counts of nodes, columns, edges, and test results synced.

Concepts¶

Concept	What it is
dbt Connection	A registered link to an Azure Blob Storage container that holds your dbt artifacts. Credentials live in Azure Key Vault.
Artifact source	The storage backend (Azure Blob) from which artifacts are downloaded on sync.
Sync Run	A single execution of the sync operation. Produces a versioned snapshot of all nodes, columns, edges, and test results.
dbt Node	One model, source, seed, or snapshot from your dbt project, identified by its `unique_id`.
Lineage edge	A directed dependency between two nodes (upstream → downstream), derived from `parent_map` in `manifest.json`.
Test result	The outcome of one dbt test execution — status, failure count, message, and execution time — tied to a specific sync run.

Setup — what you need once¶

Prereq	Where	Why
dbt project generating artifacts	Your own infrastructure	Datahub reads pre-built artifacts; it does not run dbt itself.
Azure Blob Storage	Your Azure subscription	The artifact source. One container per dbt project is a common pattern.
App Registration	Azure Active Directory	Datahub authenticates with a client ID + secret; the secret is stored in Key Vault.
Azure Key Vault	Configured on the Datahub platform	Connection secrets never touch the Datahub database.

Setting it up, end to end

dbt on Azure Blob Storage is the step-by-step walkthrough — uploading artifacts (Azure CLI, GitHub Actions, Azure DevOps), the connection form fields, testing the connection, running your first sync, and troubleshooting empty nodes / lineage / tests.