How dbt Works: Compile, Run, and the DAG

Beginner model: dbt run just "runs my SQL files."

Production model: dbt run parses the whole project, resolves every model's Jinja into plain SQL, builds a dependency graph from those resolved references, sorts that graph into a safe execution order, and only then sends each model's compiled SQL to the warehouse — one statement at a time, in dependency order, not all at once and not in file order.

1. READ    — parse every .sql file and every YAML config in the project
2. COMPILE — resolve Jinja (ref(), source(), macros) into plain SQL per model
3. GRAPH   — build the DAG from resolved dependencies, topologically sort it
4. EXECUTE — run each model's compiled SQL against the warehouse, in that order
             (CREATE TABLE AS SELECT / CREATE OR REPLACE VIEW AS, per model)

$ dbt run
Running with dbt=1.8.0
Registered adapter: snowflake=1.8.0

# ── phases 1 & 2: read + compile happen here, before any output ──
Found 6 models, 14 tests, 2 sources, 0 exposures, 0 metrics

# ── phase 3: DAG built, topological sort computed (silent, internal) ──

# ── phase 4: execution, in dependency order ──
1 of 6 START sql view model staging.stg_orders ................ [RUN]
1 of 6 OK created sql view model staging.stg_orders ........... [CREATE VIEW in 0.52s]
2 of 6 START sql view model staging.stg_customers .............. [RUN]
2 of 6 OK created sql view model staging.stg_customers ......... [CREATE VIEW in 0.48s]
3 of 6 START sql view model staging.stg_payments ................ [RUN]
3 of 6 OK created sql view model staging.stg_payments ........... [CREATE VIEW in 0.51s]
4 of 6 START sql table model marts.fct_daily_revenue ............ [RUN]
4 of 6 OK created sql table model marts.fct_daily_revenue ....... [CREATE TABLE in 1.91s]
5 of 6 START sql table model marts.dim_customers ................ [RUN]
5 of 6 OK created sql table model marts.dim_customers ........... [CREATE TABLE in 1.22s]
6 of 6 START sql table model marts.fct_payment_summary ........... [RUN]
6 of 6 OK created sql table model marts.fct_payment_summary ...... [CREATE TABLE in 0.97s]

Finished running 6 view models, 3 table models in 0 hours 0 minutes and 5.61 seconds.

Completed successfully

Done. PASS=6 WARN=0 ERROR=0 SKIP=0 TOTAL=6

select
    order_date,
    sum(amount_usd) as total_revenue_usd,
    count(distinct order_id) as order_count
from {{ ref('stg_orders') }}
where order_status = 'completed'
group by order_date

select
    order_date,
    sum(amount_usd) as total_revenue_usd,
    count(distinct order_id) as order_count
from "analytics"."dbt_prod"."stg_orders"
where order_status = 'completed'
group by order_date

select
    region,
    {{ dbt_utils.star(from=ref('stg_orders'), except=['order_id', 'region']) }}
from {{ ref('stg_orders') }}
group by region

select
    region,
    customer_id,
    order_status,
    order_placed_at,
    amount_usd
from "analytics_staging"."stg_orders"
group by region

sources:
  - name: freshcart_raw
    database: raw
    schema: freshcart_raw
    tables:
      - name: orders
      - name: customers
      - name: payments

sources:
  freshcart_raw.orders  ──┐
  freshcart_raw.customers ┼──> stg_orders ──┐
                          │                  │
                          └──> stg_customers ┼──> fct_daily_revenue
                                              │
                          freshcart_raw.payments ──> stg_payments ──┘

Edges point in the direction of "depends on".
fct_daily_revenue cannot run until stg_orders, stg_customers,
and stg_payments have all run successfully.
stg_orders cannot run until freshcart_raw.orders (a source, not
a model dbt built) exists in the warehouse.

Nodes and their dependencies:
  stg_orders          depends on: freshcart_raw.orders (source)
  stg_customers        depends on: freshcart_raw.customers (source)
  stg_payments         depends on: freshcart_raw.payments (source)
  fct_daily_revenue    depends on: stg_orders, stg_customers, stg_payments

Step 1: sources have no dbt-managed dependencies -> considered "ready" immediately
Step 2: stg_orders, stg_customers, stg_payments each depend only on a
        source -> all three become ready as soon as their source exists
Step 3: fct_daily_revenue depends on all three staging models -> only
        becomes ready once ALL THREE have completed successfully

One valid execution order dbt might compute:
  1. stg_orders
  2. stg_customers
  3. stg_payments
  4. fct_daily_revenue

Because stg_orders, stg_customers, and stg_payments share no
dependency on each other, dbt can also run them concurrently
(bounded by the --threads setting) rather than strictly sequentially --
the topological sort guarantees a valid ORDER, not that everything
must run one at a time.

1 of 4 OK created sql view model staging.stg_orders ......... [CREATE VIEW in 0.51s]
2 of 4 OK created sql view model staging.stg_customers ........ [CREATE VIEW in 0.44s]
3 of 4 ERROR creating sql table model marts.fct_daily_revenue .. [ERROR in 0.88s]
4 of 4 SKIP relation marts.fct_customer_ltv ..................... [SKIPPED]

Completed with 1 error and 1 skip:

Database Error in model fct_daily_revenue (models/marts/fct_daily_revenue.sql)
  Numeric value 'N/A' is not recognized

Done. PASS=2 WARN=0 ERROR=1 SKIP=1 TOTAL=4

# 1. A simple cron entry on a server with dbt Core installed
0 6 * * * cd /opt/freshcart_dbt && dbt run >> /var/log/dbt_run.log 2>&1

# 2. An Airflow task (conceptually — exact operator syntax varies)
dbt_run_task = BashOperator(
    task_id="run_dbt_models",
    bash_command="cd /opt/freshcart_dbt && dbt run",
)
# scheduled via the surrounding Airflow DAG's schedule_interval

# 3. dbt Cloud's built-in scheduler — configured entirely in the UI,
#    no external cron or Airflow needed; dbt Cloud runs its own
#    hosted dbt Core execution on the schedule you configure

-- models/marts/fct_daily_revenue.sql
{{ config(materialized='table') }}

select
    order_date,
    sum(amount_usd) as total_revenue_usd
from {{ ref('stg_orders') }}
group by order_date

{{
  config(
    materialized='incremental',
    unique_key='event_id'
  )
}}

select
    event_id,
    user_id,
    event_type,
    event_at
from {{ ref('stg_events') }}

{% if is_incremental() %}
  where event_at > (select max(event_at) from {{ this }})
{% endif %}

sources:
  - name: freshcart_raw
    database: raw
    schema: freshcart_raw
    tables:
      - name: orders

select
    ord_id                              as order_id,
    cust_id                             as customer_id,
    ord_stat                            as order_status,
    cast(ord_placed_ts as timestamp)    as order_placed_at,
    amt_cents / 100.0                   as amount_usd
from {{ source('freshcart_raw', 'orders') }}
where is_deleted = false

{{ config(materialized='table') }}

select
    date_trunc('day', order_placed_at) as order_date,
    sum(amount_usd)                    as total_revenue_usd,
    count(distinct order_id)           as order_count
from {{ ref('stg_orders') }}
where order_status = 'completed'
group by 1

select
    ord_id                              as order_id,
    cust_id                             as customer_id,
    ord_stat                            as order_status,
    cast(ord_placed_ts as timestamp)    as order_placed_at,
    amt_cents / 100.0                   as amount_usd
from "raw"."freshcart_raw"."orders"
where is_deleted = false

select
    date_trunc('day', order_placed_at) as order_date,
    sum(amount_usd)                    as total_revenue_usd,
    count(distinct order_id)           as order_count
from "analytics_staging"."stg_orders"
where order_status = 'completed'
group by 1

$ dbt run --select stg_orders fct_daily_revenue
Running with dbt=1.8.0

1 of 2 START sql view model analytics_staging.stg_orders ..... [RUN]
1 of 2 OK created sql view model analytics_staging.stg_orders  [CREATE VIEW in 0.61s]
2 of 2 START sql table model analytics.fct_daily_revenue ...... [RUN]
2 of 2 OK created sql table model analytics.fct_daily_revenue  [CREATE TABLE in 1.84s]

Completed successfully

Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

At Airbnb:

An analytics engineer on a listings-quality team makes a change to a shared staging model and, before merging, wants to see exactly what SQL will be generated for the several downstream mart models that reference it — without touching the warehouse and without waiting for a full dbt run against a large project. Running dbt compile --select stg_listings+ and reading the generated files under target/compiled/ lets them verify the join logic resolved correctly in seconds, entirely safely, before ever executing anything for real.

At JetBlue:

A data platform team debugging a nightly pipeline failure sees that a downstream flight-delay mart model failed with a "relation does not exist" error. Because they understand the DAG and topological sort, they immediately check whether the specific staging model it depends on actually succeeded earlier in the same run, rather than assuming the mart model's own SQL is broken — and they find the real cause: a --select flag in that night's CI job had accidentally excluded the staging model from the run entirely.

At Faire:

A team migrating a large events table to an incremental materialization to cut nightly run time discovers, a few weeks later, that a small number of late-arriving events — records that show up in the source system a day or two after their actual event timestamp — are being permanently missed, because the incremental filter only looked forward from the last processed timestamp and never re-checked a trailing window. Understanding exactly how the is_incremental() logic and the {{ this }} reference work, as covered in Part 06, is what let the team diagnose this precisely as a filter-logic problem and fix it by widening the incremental window with a small lookback buffer, rather than reverting to a full table rebuild out of caution.

In a system design / analytics-engineering interview:

A candidate is asked what would happen if dbt tried to run a mart model before its staging dependency finished successfully. A weak answer says "dbt handles that automatically" without explaining how. A strong answer explains the actual mechanism — the DAG built from resolved ref() calls, the topological sort computed from that graph, and the guarantee that a model is never scheduled for execution until every node it depends on has completed — which is precisely the depth covered in Part 03 and Part 04.

1. Walk me through exactly what happens when you run `dbt run`.

As Part 01 covers step by step, dbt first reads every .sql file in the project's models/ directory along with the project's YAML configuration. For each model, it resolves any Jinja templating — most importantly ref() calls (references to other models) and source() calls (references to raw tables) — into the fully-qualified, environment-correct table names those functions point to, producing plain SQL.

From those resolved references, dbt builds a dependency graph — the DAG, covered in Part 03 — where each model is a node and each ref()/source() call is a directed edge. It then performs a topological sort on that graph to compute a valid execution order: every model only appears after everything it depends on. Finally, dbt executes each compiled SQL statement against the warehouse in that order, typically as a CREATE TABLE AS SELECT or CREATE OR REPLACE VIEW AS, depending on how the model is materialized.

2. What is the difference between `dbt compile` and `dbt run`, and when would you use compile on its own?

dbt compile performs only the first half of what dbt run does: it resolves Jinja templating and ref()/source() calls into plain SQL and writes the result to files under target/compiled/, but it never sends any of that SQL to the warehouse. Nothing is created, updated, or queried. dbt run does that same compilation step and then actually executes the compiled SQL against the warehouse.

As Part 02 explains, this makes dbt compile a debugging tool: if you're not sure exactly what SQL your Jinja is generating — say, a macro isn't producing the join condition you expect — you run dbt compile, open the corresponding file in target/compiled/, and read the literal SQL that would have been run, without touching the warehouse or waiting on a real execution.

3. Explain the DAG in dbt — what are the nodes, what are the edges, and what guarantee does it give you?

As covered in Part 03, the DAG's nodes are the objects dbt manages: models, sources, seeds (static CSV data loaded by dbt), and snapshots (point-in-time captures of slowly changing data). Its edges are the dependencies declared through ref() calls (model-to-model) and source() calls (model-to-raw-source). The graph is acyclic by construction — dbt will raise a compilation error if you accidentally create a circular dependency, because a cycle would make a valid execution order impossible to compute.

The guarantee the DAG gives you, enforced through the topological sort described in Part 04, is that a model will never be run until every model and source it depends on has already completed successfully. This is what lets you build a three-layer pipeline — staging models on top of raw sources, then intermediate or mart models on top of staging — and trust that dbt will never try to build the mart model before its staging dependencies exist.

4. If dbt doesn't schedule itself, how does it actually run on a recurring basis in production?

dbt Core, run from the command line, only does anything the moment you invoke it — it has no concept of "run this every day at 6am" built into the tool itself. As Part 05 covers, something external has to trigger it on that schedule. Common options are a cron job on a server that shells out to dbt run, an Airflow DAG with a task that triggers a dbt run (often via a dedicated operator), a scheduled CI/CD pipeline (a GitHub Actions workflow on a cron trigger, for instance), or dbt Cloud's own built-in job scheduler, which is one of the main things you're paying for when you use dbt Cloud instead of self-managing dbt Core's scheduling yourself.

This is also why "orchestration" and "transformation" are treated as separate concerns in a modern data stack: dbt's job is only to correctly transform data once triggered, in the right dependency order; deciding when and how often that trigger fires is a scheduler's job, sitting one layer above dbt.

5. What is the difference between an incremental model and a regular table model, and when would you reach for one over the other?

As Part 06 covers, a table materialization fully recomputes its entire result set from the SELECT statement on every single run, discarding and rebuilding the table from scratch. An incremental model instead only fully rebuilds on its very first run; on every subsequent run, it computes a much narrower query — typically filtered to only new or changed rows — and merges just those rows into the table that already exists, using the special is_incremental() Jinja check and a {{ this }} reference back to the model's own current state.

The trade-off is complexity versus cost: a table materialization is simpler to reason about — the model's output is always exactly what the SELECT would produce, in full, right now — while an incremental model requires more careful logic (choosing a correct incremental filter, handling late- arriving or updated historical rows) in exchange for dramatically lower compute cost on very large fact tables where reprocessing the full history on every run would be prohibitively slow or expensive. In an interview, the strongest answer names both the mechanism and the trade-off, rather than treating "incremental" as simply the fast option to always prefer.

6. Trace a three-model chain — a raw source, a staging model, and a mart model — through compilation and execution.

As worked through in detail in Part 07: a raw orders table lands via ingestion into a raw schema; it is declared in dbt as a source(), not a model, because dbt did not create it. A staging model, stg_orders, contains select ... from {{ source('freshcart_raw', 'orders') }}, doing light cleaning — renaming columns, casting types, filtering out soft-deleted rows. A mart model, fct_daily_revenue, contains select ... from {{ ref('stg_orders') }} group by ..., aggregating the cleaned staging data into a business-ready table.

At compile time, dbt resolves the source() call in stg_orders into the literal raw.freshcart_raw.orders table reference, and resolves the ref() call in fct_daily_revenue into the literal, environment-correct name of the stg_orders table or view. The DAG built from those two edges guarantees stg_orders is fully built before fct_daily_revenue even starts, and at runtime dbt executes the compiled CREATE VIEW stg_orders AS ... first, then the compiled CREATE TABLE fct_daily_revenue AS ... second.

Hardcoding schema.table names instead of using ref() and source()

Writing `select * from raw.freshcart_raw.orders` directly inside a model, instead of `select * from {{ source('freshcart_raw', 'orders') }}`, breaks dbt's ability to build an accurate dependency graph and means the hardcoded name will silently be wrong in a different environment (dev vs prod schemas). See Part 01 and Part 03.

Assuming dbt compile actually changes anything in the warehouse

Running dbt compile to "test" a model and then being surprised nothing changed in the warehouse — it is not supposed to. Compile only resolves Jinja and writes plain SQL to target/compiled/; you need dbt run to actually execute anything. See Part 02.

Expecting dbt to detect and run only downstream models automatically after a change, with no selector

By default, `dbt run` with no arguments runs every model in the project, not just ones affected by your recent edit. Beginners are sometimes surprised a `dbt run` after touching one file takes as long as a full run. Selecting a subset (covered alongside DAG mechanics in Part 04) requires an explicit --select flag with the right graph operator (e.g. `+model_name` to include everything upstream of it).

Creating an accidental circular dependency between two models

Model A references model B with ref(), and model B also references model A with ref(). This makes a valid execution order impossible to compute, and dbt raises a compilation error refusing to build the DAG rather than guessing. See Part 03.

Believing a scheduled Airflow DAG or cron job IS "dbt's scheduler"

Conflating the orchestration layer (whatever triggers dbt on a schedule) with dbt itself. dbt Core has no concept of a schedule; the trigger is always external, whether that's Airflow, cron, CI, or dbt Cloud's own scheduler. See Part 05.

Treating an incremental model's filter logic as a minor detail rather than the core of its correctness

Copying an incremental model's is_incremental() filter from another project without checking whether late-arriving or updated rows in your own data need a lookback window. Getting this filter wrong doesn't cause a visible error — it silently and permanently drops rows that never get reprocessed. See Part 06.

`Compilation Error: Found a cycle: model.freshcart.model_a --> model.freshcart.model_b --> model.freshcart.model_a`

This means two (or more) models reference each other through ref(), directly or via a longer chain, making a valid topological sort impossible. Fix it by restructuring the logic so the dependency only flows one direction — usually one of the two models has logic that actually belongs upstream of the other, not alongside it.

`Database Error: Relation "STG_ORDERS" does not exist` on a model that clearly ran earlier in the same dbt run

This usually means the earlier model was materialized as a view in a schema the current model's connection cannot see, or an intermittent warehouse replication delay between the CREATE and the subsequent SELECT (more common on some warehouses under specific isolation settings). Confirm the model actually succeeded in dbt's run output before assuming it is a dependency-ordering bug.

`dbt run` completes successfully but a downstream model still shows stale data

If the downstream model is materialized as a table (not a view) and something skipped a dependency's rebuild — for example you ran `dbt run --select stg_orders` and forgot the downstream model also needed rebuilding — the table simply was not touched this run. Check exactly which models were selected in the run's output list against what you expected.

`Compilation Error: 'source' is undefined` inside a model

This is a Jinja/templating error, not a database error — it happens when {{ source(...) }} is called somewhere dbt's Jinja context does not have that function available, most often because of a syntax mistake elsewhere in the file (an unclosed {% %} block above it) that breaks how the rest of the file is parsed. Check the file for a stray or missing Jinja tag before the failing line.

A scheduled dbt Cloud job or CI job simply never triggers on the expected day/time

This is almost never a dbt bug — it is a scheduler configuration issue (timezone mismatch in a cron expression, an Airflow DAG that is paused, a dbt Cloud job schedule left disabled after being edited). Since dbt itself has no concept of a schedule (Part 05), any "it didn't run when expected" issue lives entirely in whatever triggers it, not in dbt's own logic.

`Database Error: Duplicate row detected during DML operation` on an incremental model's merge step

This happens when the unique_key configured for an incremental model does not actually uniquely identify rows in the incoming data — the underlying MERGE statement dbt generates cannot decide which incoming row should update which existing row when more than one incoming row shares the same key. Check that the chosen unique_key genuinely has no duplicates in the source data, per run.

3 of 6 OK created sql view model staging.stg_orders ......... [CREATE VIEW in 0.49s]
4 of 6 ERROR creating sql table model marts.fct_daily_revenue .. [ERROR in 0.71s]

Database Error in model fct_daily_revenue (models/marts/fct_daily_revenue.sql)
  Numeric value 'N/A' is not recognized

select
    order_date,
    sum(amount_usd) as total_revenue_usd,
    count(distinct order_id) as order_count
from "analytics_staging"."stg_orders"
where order_status = 'completed'
group by 1