CI/CD for dbt Projects
Why dbt projects need continuous integration just like application code, the Slim CI pattern with state:modified+ and --defer, a real GitHub Actions workflow, dbt Cloud's built-in CI jobs versus self-hosting, and a full PR-to-production deployment flow.
A dbt Project Is Application Code, and It Deserves the Same Discipline
It is easy to treat a dbt project as "just SQL" and skip the engineering rigor that a backend service would get by default — a pull request review, an automated test run, a deployment gate. That instinct is wrong, and it gets more expensive to fix the longer a team waits. A dbt project is a compiled, versioned, dependency-graphed piece of software that runs directly against production data. A broken model does not throw a stack trace in a staging environment nobody looks at — it silently corrupts a table that a VP's dashboard reads from tomorrow morning.
Continuous integration for a dbt project means one specific, concrete thing: on every pull request that changes a model, test, or macro, an automated job builds the changed models (and only the changed models, plus whatever depends on them) in an isolated environment, runs their tests, and reports pass or fail directly on the PR before a human ever has to eyeball a diff of raw SQL and guess whether it compiles correctly against real data.
Without CI: a model change is reviewed as text, merged on faith, and its first real execution against production data happens on the next scheduled run — which is also the first moment anyone finds out it references a column that was renamed three weeks ago.
With CI: the exact same change is built and tested against a fresh, isolated set of tables before merge. A broken ref(), a failing not_null test, or a SQL compilation error shows up as a red check on the PR, not as a 3am page.
The cost of skipping this is not hypothetical. A single incremental model with an off-by-one window in its is_incremental() filter can silently double-count revenue for weeks before anyone notices a dashboard total looks slightly too high. CI does not eliminate every bug — nothing does — but it eliminates the entire class of bug that a fresh build against real, reasonably-sized data would have caught immediately.
The Obvious First Attempt: Run "dbt build" on Every PR
The most obvious way to add CI to a dbt project is to run dbt build — building every single model and running every single test in the entire project — on every pull request. This works, and for a genuinely small project (a few dozen models), it is a perfectly reasonable place to start.
It stops working as the project grows. A 300-model warehouse with several large incremental fact tables can take 40 minutes or more to build from scratch. A one-line change to a single staging model — renaming a column, fixing a typo in a comment — now triggers a 40-minute CI run that rebuilds 299 models that were not touched at all. Multiply that by a team merging a dozen PRs a day and CI becomes a bottleneck that slows down the exact process it was meant to speed up.
| Approach | What it builds | Cost at 300+ models |
|---|---|---|
| Full dbt build on every PR | Every model in the entire project, regardless of what changed. | Extremely slow, expensive warehouse compute charged on every PR, does not scale with team size. |
| Slim CI (state:modified+) | Only models that changed, plus everything downstream of them. | Scales with the size of the change, not the size of the project — a one-model PR builds roughly one model. |
The fix is not a faster warehouse or a bigger CI runner — it is building less. dbt has a built-in mechanism for figuring out exactly which models a given change actually affects, and building only those. That mechanism is state comparison, and the pattern built on top of it is what the dbt community calls Slim CI.
manifest.json: dbt's Compiled Picture of Your Project
Every time dbt parses and compiles a project — for dbt run, dbt build,dbt compile, or any other invocation — it writes a file called manifest.json into the target/ directory. This file is dbt's complete, structured representation of the project at that moment: every model, its compiled SQL, its resolved ref() and source() dependencies, every test, every macro, every column-level and model-level config, and a content hash of each file.
The manifest is not a debugging artifact you are meant to read by hand — it is the mechanism that makes Slim CI possible at all. dbt can compare two manifests — one from a previous, known-good state (typically the last successful production run) and one from the current, in-review state — and compute a precise diff: which model definitions actually changed, which tests changed, which sources changed. That diff is what state:modified means.
# Any dbt invocation that parses the project writes target/manifest.json
dbt compile
# or
dbt run
# or
dbt build
# The file that matters for CI:
ls target/manifest.jsonWhere the production manifest has to live for CI to see it
For a CI run to compare against production state, it needs a copy of production's manifest.json available at CI time — and a CI runner is a fresh, ephemeral environment with no memory of any previous run. This means the production manifest has to be persisted somewhere the CI job can fetch it from before comparing state. The most common pattern is uploading the manifest to an object storage bucket (an S3 bucket, a GCS bucket, an Azure Blob container) as the last step of every successful scheduled production run, then downloading that same file as the first step of every CI run.
# Last step of the scheduled PRODUCTION run (after a successful dbt build):
aws s3 cp target/manifest.json s3://my-company-dbt-artifacts/prod/manifest.json
# First step of every CI run, before comparing state:
aws s3 cp s3://my-company-dbt-artifacts/prod/manifest.json ./prod-manifest/manifest.jsonstate:modified+ — Build Only What Changed, Plus Everything Downstream
With a production manifest available for comparison, dbt's state: selector method lets you select models based on how they differ from that reference state. state:modified selects every model whose definition changed relative to the comparison manifest — a different compiled SQL body, a changed config, a changed test. The trailing + is dbt's graph operator meaning "and everything downstream of these" — every model that directly or transitively depends on a modified model via ref().
dbt build --select state:modified+ --state ./prod-manifestRead literally: build every model that changed, and every model downstream of a changed model, compared against the manifest found in ./prod-manifest. A PR that only edits a single staging model near the bottom of the DAG builds that staging model plus its downstream marts — typically a handful of models, not the entire 300-model project. A PR that edits a widely-referenced core model (a shared dim_customers, say) correctly builds a much larger slice of the DAG, because a change that far upstream genuinely could affect everything downstream of it — Slim CI is precise, not naively small.
| Selector | What it means | When to use it |
|---|---|---|
| state:modified | Only models whose own definition changed — not their downstream dependents. | Rare on its own; usually paired with + to also catch downstream impact. |
| state:modified+ | Modified models plus everything downstream of them. | The standard Slim CI selector for a pull-request build. |
| state:new | Models that exist in the current project but not in the comparison state at all. | Distinguishing a genuinely new model from a modified existing one, useful in more advanced CI reporting. |
| +state:modified | Modified models plus everything upstream that feeds them. | Less commonly needed for CI — upstream models did not change, so rebuilding them adds cost without validating anything new. |
ref() and source() rather than hardcoded table names — exactly the DAG construction covered in the models-basics and sources-and-ref modules. A project with raw, hardcoded table references anywhere in its model SQL has holes in its dependency graph that dbt cannot see, and state comparison silently misses any downstream impact through that hole.--defer: Let CI Reference Already-Built Production Tables It Did Not Rebuild
Slim CI's state:modified+ selector solves what to build. It does not, by itself, solve a second problem: a downstream model that was correctly excluded from this PR's build still has a ref() pointing at an upstream model — and if that upstream model was also excluded (because it did not change), the downstream model's query will fail, because the table it references was never created in this CI run's isolated schema at all.
--defer, used together with --state, is exactly the fix. It tells dbt: "for any model this run did not build itself, resolve its ref() to the already-built production table instead of failing because it does not exist in this run's schema." Combined with state:modified+, this means a CI run builds a small, changed slice of the DAG in a fresh, isolated schema, while every unbuilt upstream dependency is deferred to production — producing correct, realistic results without rebuilding the entire warehouse.
dbt build \
--select state:modified+ \
--state ./prod-manifest \
--defer \
--favor-state--favor-state (available in newer dbt versions alongside --defer) makes the deferral behavior even more predictable: it tells dbt to always prefer the production version of an unselected node over any same-named object that might happen to already exist in the CI schema, removing an edge case where a stale leftover object from a previous CI run could otherwise get picked up by mistake.
Worked example — what actually happens for a one-model PR
DAG: stg_orders -> int_orders_joined -> fct_orders -> rpt_revenue_daily
PR changes only stg_orders.sql (added a new column).
dbt build --select state:modified+ --state ./prod-manifest --defer
state:modified = { stg_orders } (its definition changed)
state:modified+ = { stg_orders, int_orders_joined, fct_orders, rpt_revenue_daily }
(everything downstream, since a column added upstream could
affect all four models' correctness)
This CI run builds all four models fresh, in an isolated CI schema.
Any OTHER model in the 300-model project that these four do not depend on
is never touched -- and if any of these four happen to depend on some
fifth, unrelated model that did NOT change, --defer resolves that ref()
straight to the real production table instead of failing.Two Ways to Run This: dbt Cloud's Built-In CI Job, or Your Own Pipeline
Everything in Parts 03 through 05 — comparing state, running state:modified+, deferring to production — is a set of dbt features, not a specific CI vendor's feature. You can wire it up yourself in GitHub Actions, GitLab CI, CircleCI, or Jenkins. Or, if the project runs on dbt Cloud, you can use its purpose-built CI job type, which implements this entire pattern out of the box.
| Aspect | dbt Cloud CI job | Self-hosted CI (GitHub Actions, etc.) |
|---|---|---|
| Manifest storage and retrieval | Handled automatically — dbt Cloud tracks every job's artifacts and knows which one is the production comparison state. | You build this yourself — typically an S3/GCS upload step after every production run, and a download step at the start of CI. |
| Triggering | Automatically runs on every PR against a configured branch, with zero pipeline YAML to write. | You write and maintain the workflow file yourself (Part 07 below is a full example). |
| Ephemeral schema creation | Automatically builds into a temporary, PR-specific schema and tears it down after. | You configure this via profiles.yml / environment variables, usually keyed off a CI-provided PR number. |
| Status reporting on the PR | Native GitHub/GitLab check integration, built in. | You wire this up via the CI platform's own PR status API, usually already provided by the CI runner itself. |
| Cost | Requires a dbt Cloud plan that includes CI jobs. | Only warehouse compute plus whatever your existing CI runner already costs — no separate dbt-specific fee. |
| Control and portability | Tied to dbt Cloud's job model and UI. | Fully owned pipeline-as-code, portable across CI vendors, easier to customize with org-specific steps. |
Neither option is a strictly better engineering choice — they are the same trade-off any managed-versus-self-hosted decision presents. A small team already paying for dbt Cloud gets Slim CI essentially for free with no pipeline to maintain. A team that is already deeply invested in its own CI platform, or that needs fine-grained custom steps (a security scan, a custom cost-estimation step before build, posting to a specific Slack channel), often prefers owning the pipeline directly.
A Complete, Working Slim CI Workflow in GitHub Actions
Putting Parts 03 through 06 together into one working pipeline. This workflow triggers on every pull request, fetches the production manifest from S3, installs dbt and project dependencies, then runs the Slim CI build with defer, reporting pass or fail as a required check on the PR.
name: dbt Slim CI
on:
pull_request:
paths:
- 'models/**'
- 'macros/**'
- 'tests/**'
- 'seeds/**'
- 'dbt_project.yml'
- 'packages.yml'
jobs:
slim-ci:
runs-on: ubuntu-latest
environment: ci
steps:
- name: Check out the PR branch
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Install dbt and adapters
run: pip install dbt-core==1.8.0 dbt-snowflake==1.8.0
- name: Install dbt packages
run: dbt deps
- name: Configure AWS credentials for manifest storage
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-east-1
- name: Fetch production manifest for state comparison
run: |
mkdir -p prod-manifest
aws s3 cp s3://my-company-dbt-artifacts/prod/manifest.json ./prod-manifest/manifest.json
- name: Run Slim CI build
env:
DBT_SNOWFLAKE_ACCOUNT: ${{ secrets.DBT_SNOWFLAKE_ACCOUNT }}
DBT_SNOWFLAKE_USER: ${{ secrets.DBT_SNOWFLAKE_USER }}
DBT_SNOWFLAKE_PASSWORD: ${{ secrets.DBT_SNOWFLAKE_PASSWORD }}
DBT_TARGET_SCHEMA: "pr_${{ github.event.pull_request.number }}"
run: |
dbt build \
--select state:modified+ \
--state ./prod-manifest \
--defer \
--favor-state \
--target ci
- name: Post build results summary
if: always()
run: cat target/run_results.json | python -m json.tool | head -n 40Two details worth calling out explicitly. First, the paths: filter on the trigger means this workflow does not even start for PRs that only touch unrelated files (documentation, CI config for a different pipeline) — a small but real cost saving on top of Slim CI's own model-level selectivity. Second, the target schema name is derived from the PR number (pr_${{ github.event.pull_request.number }}), so multiple open PRs get independent, non-colliding schemas to build into, and a ci target in profiles.ymlpoints at that dynamically-named schema rather than a fixed one.
my_project:
target: ci
outputs:
ci:
type: snowflake
account: "{{ env_var('DBT_SNOWFLAKE_ACCOUNT') }}"
user: "{{ env_var('DBT_SNOWFLAKE_USER') }}"
password: "{{ env_var('DBT_SNOWFLAKE_PASSWORD') }}"
role: ci_role
database: analytics
schema: "{{ env_var('DBT_TARGET_SCHEMA') }}"
warehouse: ci_wh
threads: 4CI Is Not Deployment — a Separate, Scheduled Production Pipeline Ships the Change
A common point of confusion: the PR-time Slim CI build described in Parts 04 through 07 does not ship anything to production. It validates a proposed change in an isolated, throwaway schema, and that schema is torn down after the PR closes, whether merged or not. Production itself is updated by a completely separate process — a scheduled job, decoupled from any individual PR, that runs after a merge to the main branch.
| Pipeline | Triggered by | Builds into | Purpose |
|---|---|---|---|
| PR-time Slim CI | Opening or updating a pull request | A temporary, PR-specific schema | Validate the proposed change before a human approves merging it. |
| Production deployment job | A merge to main, or a fixed schedule (hourly, nightly) | The real production schema | Actually update the tables business users and dashboards query. |
This separation matters for a subtle reason: the production job should almost always run a full, unfiltered dbt build (or a build scoped by business logic like freshness tiers, not by what changed in git) — because production needs the entire warehouse to be internally consistent, not just the slice a particular PR touched. Slim CI's whole value proposition is speed during review; production's value proposition is completeness and consistency, which is the opposite optimization.
name: dbt Production Deploy
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- run: pip install dbt-core==1.8.0 dbt-snowflake==1.8.0
- run: dbt deps
- name: Full production build
env:
DBT_TARGET_SCHEMA: analytics
run: dbt build --target prod
- name: Publish manifest.json for the next CI run's state comparison
run: |
aws s3 cp target/manifest.json s3://my-company-dbt-artifacts/prod/manifest.json
- name: Generate and publish dbt docs
run: |
dbt docs generate --target prod
aws s3 sync target/ s3://my-company-dbt-docs/ --exclude "*" --include "*.html" --include "*.json" --include "*.js"Notice the second-to-last step: publishing the freshly-built manifest.json back to the same S3 location the Slim CI workflow reads from. This closes the loop described in Part 03 — every successful production deploy updates the "known-good" reference state that the very next PR's CI run will compare itself against, so Slim CI's notion of "what changed" always means "changed since the last thing that actually shipped," not some stale, arbitrarily old snapshot.
dbt build again after fixing the underlying issue reproduces a correct, consistent state, rather than requiring a manual rollback of half-applied changes.A Full Worked Example: One Change, Start to Finish
Tying every piece together with one concrete change moving through the whole system, end to end.
1. An analytics engineer opens a PR adding a new column to stg_orders.sql
and a corresponding not_null test in its schema.yml.
2. GitHub Actions triggers the Slim CI workflow (Part 07) because the PR
touches a file under models/.
3. CI fetches the last-published production manifest.json from S3
(published by the previous merge's production deploy job -- Part 08).
4. dbt build --select state:modified+ --state ./prod-manifest --defer runs.
dbt detects stg_orders changed, and that int_orders_joined, fct_orders,
and rpt_revenue_daily all depend on it transitively -- exactly four
models get built fresh in an isolated pr_482 schema. Every other model
in the 300-model project is skipped and deferred to production.
5. The new not_null test on stg_orders runs against the freshly built PR
schema and passes. CI reports a green check on the PR.
6. A teammate reviews the diff, sees the green Slim CI check, approves,
and merges to main.
7. The production deploy job (Part 08) triggers on the merge, running a
full dbt build --target prod across the entire project.
8. On success, the job publishes the new manifest.json back to S3 --
this becomes the comparison state for the NEXT PR's Slim CI run.
9. dbt docs are regenerated and published, so the updated column and
test are immediately visible to anyone browsing the documentation
site covered in the documentation module.Nothing in this flow required a human to manually run dbt against production, guess which models were affected, or eyeball a SQL diff without seeing it actually execute. Every step is either fully automated or a single, informed approve-and-merge decision backed by a real build result.
How Many Environments Does a dbt Project Actually Need, and What Runs Where
Everything covered so far assumes two environments — a PR's isolated CI schema and production — but most real teams settle on a three-environment model, adding a persistent staging (sometimes called "QA") environment between them. Understanding what each environment is actually for, rather than treating "more environments" as automatically safer, is itself a design decision worth being able to justify in an interview or an architecture review.
| Environment | Lifetime | What builds into it | Purpose |
|---|---|---|---|
| CI (per-PR) | Created when a PR opens, torn down when it closes. | Only state:modified+ selected models, per PR, deferred to production for everything else. | Validate one specific change before merge, at minimal cost. |
| Staging / QA | Persistent, shared, rebuilt on every merge to a staging branch. | A full or near-full dbt build, on real (often production-copy) data. | A stable place to catch integration issues across several merged changes before they reach production — problems Slim CI's narrow, per-PR scope cannot see by design. |
| Production | Persistent, the environment real dashboards and reports query. | A full dbt build, on a fixed schedule plus on merge to main. | The actual source of truth every downstream consumer relies on. |
The staging environment exists specifically to catch a class of bug Slim CI cannot: two PRs, each individually valid and each passing its own narrow Slim CI check, that conflict with each other only when both are merged — one PR renaming a column, another PR (opened before the rename merged) still referencing the old name. Neither PR's own CI run ever builds both changes together, because each Slim CI run only ever sees its own PR's diff against the last known-good production state. A shared staging environment, rebuilt from a full merge history rather than a single PR's diff, is what surfaces this kind of cross-PR conflict before it reaches production.
main -> production (scheduled + on-merge full dbt build)
staging -> staging/QA (full dbt build on every merge to this branch)
feature/* -> PR-specific Slim CI schema (torn down on PR close)
Typical flow:
1. Engineer branches feature/add-ltv-metric off staging.
2. Opens a PR back into staging -- Slim CI runs (Part 04-07).
3. Merges into staging -- triggers a full QA build; team eyeballs the
staging dashboards before promoting further.
4. A separate, deliberate promotion PR merges staging into main --
often just a fast-forward merge once QA looks correct --
triggering the full production deploy job from Part 08.Keeping environment-specific configuration out of model SQL
None of this three-tier structure should ever require a model's SQL to branch on which environment it is running in. target.name (available in Jinja) lets a model or macro behave differently per environment when genuinely necessary — most commonly to sample a smaller subset of data outside of production, exactly the local/CI distinction Part 07 of this module's system-design companion module discusses — but reaching for it routinely is usually a sign a materialization or variable-based config would be the better tool.
{{
config(
materialized='table'
)
}}
SELECT *
FROM {{ source('app', 'events') }}
{% if target.name != 'prod' %}
-- keep non-production builds fast and cheap by sampling
WHERE random() < 0.05
{% endif %}Handling Warehouse Credentials and Secrets Safely in a CI Pipeline
Every CI run in this module's examples needs live warehouse credentials to actually connect and build models — which means a dbt CI pipeline is also a place real production-adjacent secrets flow through automation triggered by, in the case of a public or externally-contributed repository, code a stranger wrote. Getting this wrong is not a hypothetical risk; it is one of the more common ways a CI pipeline becomes a security incident rather than a productivity tool.
A dedicated CI role, never a personal or admin credential
The CI pipeline should authenticate as its own dedicated warehouse role — commonly named something like ci_role, as used in this module's Part 07 example — with privileges scoped to exactly what CI needs: creating and dropping PR-specific schemas, reading from source tables, and nothing more. It should never authenticate as an individual engineer's personal credentials, and it should never use a broadly-privileged administrative role, because a compromised or misconfigured CI job then has only as much blast radius as that narrow role permits.
| Credential scope | Risk if compromised | Why the narrower option is correct for CI |
|---|---|---|
| A personal engineer credential reused for CI | Inherits that specific person's full warehouse access, and rotating it (if the engineer leaves) silently breaks CI. | CI's identity should not be tied to any one person's employment status or personal permission set. |
| A broad admin/owner role | A compromised CI job could read, modify, or drop anything in the warehouse, far beyond what dbt build actually needs. | The role a pipeline runs as should never exceed what that pipeline's own job actually requires. |
| A dedicated, narrowly-scoped ci_role | Limited to schema create/drop within a CI-designated namespace and read access to sources — a compromise here cannot reach unrelated data. | This is the role CI should actually run as: exactly enough privilege, no more. |
Secrets belong in the CI platform's secret store, never in a workflow file
Every credential referenced in this module's GitHub Actions example — the Snowflake account, user, and password — is pulled from secrets.*, GitHub Actions' encrypted secret store, never hardcoded into the YAML file itself. A workflow file lives in the same git history as everything else in the repository, including any fork of it; a credential committed directly into that file is effectively public the moment the repository is, or the moment anyone with read access to the repo's history looks.
Least-privilege schema access for the PR-specific CI schema
Because Part 07's CI schema naming is dynamic (derived from the PR number), the CI role's grant should be scoped to a pattern, not a single fixed schema — but that pattern should still be as narrow as practical, typically a dedicated CI-reserved database or schema prefix that has no overlap with where real production tables live, so that even a misconfigured or buggy CI run touching the wrong schema name cannot accidentally reach a production object.
CREATE ROLE ci_role;
GRANT USAGE ON WAREHOUSE ci_wh TO ROLE ci_role;
GRANT USAGE ON DATABASE analytics TO ROLE ci_role;
-- CI may create/drop its own PR-specific schemas, scoped by a reserved prefix
GRANT CREATE SCHEMA ON DATABASE analytics TO ROLE ci_role;
-- CI may read raw source tables, needed to build from them, but cannot
-- write to or drop anything outside schemas it created itself
GRANT USAGE ON SCHEMA analytics.raw_sources TO ROLE ci_role;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics.raw_sources TO ROLE ci_role;
-- Deliberately NOT granted: any privilege on the production-facing
-- analytics.marts schema itself -- CI only ever reads production data
-- via --defer's ref() resolution, at query time, never writes to it.What Happens When the Production Deploy Job Fails, and How to Roll Back Safely
Everything covered through Part 11 makes a PR-time failure cheap and low-stakes — a red check, no merge, nothing shipped. A production deploy job failure is a different animal entirely: by the time it runs, the change already merged to main, and a failure there means either a build that did not complete (leaving some tables stale but not obviously broken) or, worse, a build that completed but produced silently wrong data because a test that should have caught the problem did not.
The three distinct production-failure shapes, and why each needs a different response
| Failure shape | What actually happened | Correct response |
|---|---|---|
| A model fails to compile or a warehouse error occurs mid-build | The build stops partway through; some models rebuilt successfully, others did not run at all, leaving stale (not wrong) data for whatever did not get to. | Fix the underlying issue and rerun the full dbt build target=prod — idempotent rebuilding means simply running it again produces a correct end state, no manual table-by-table rollback needed. |
| A test fails after the model successfully rebuilds | The new table is built and is live, but a not_null, unique, or business-rule test caught something wrong with the data itself. | This is the most urgent case — the wrong data is already live and queryable. The immediate response is reverting the merge that introduced the change (a git revert, not just fixing forward) and rerunning production from the reverted state, since the priority is getting correct data live again quickly, with root-causing done afterward. |
| The build and every test pass, but a stakeholder later reports a wrong number the tests did not catch | A genuine test-coverage gap — the assertion that would have caught this simply did not exist. | Fix the underlying model AND add the missing test in the same PR, so the specific gap that let this through is closed permanently, not just patched for this one instance. |
The middle row is the one worth internalizing most deeply for an interview: a failed test after a successful rebuild means wrong data is already live in production, which is a fundamentally more urgent situation than a build that simply did not finish. Treating both failure shapes the same way — "something is red, go investigate calmly" — misses that one of them has customer-facing wrong numbers live right now and the other does not.
# The moment a production test failure is confirmed to mean live wrong data:
git revert <merge-commit-sha>
git push origin main
# This retriggers the production deploy job (Part 08) automatically,
# rebuilding from the last known-good state before the problematic merge.
# Root-causing the actual bug happens AFTERWARD, in a fresh PR,
# validated by Slim CI same as any other change -- not under the
# pressure of live wrong data still being served.Alerting on a production failure, not just logging it
A production dbt build failure that only shows up in a CI dashboard nobody is actively watching is barely better than no monitoring at all. The production deploy job should page or notify the owning team directly and immediately on any non-zero exit code — commonly by adding a final, always-run step that posts to a Slack channel or triggers a paging tool on failure, distinct from the routine success notifications a scheduled job might otherwise send quietly.
- name: Full production build
id: dbt_build
env:
DBT_TARGET_SCHEMA: analytics
run: dbt build --target prod
- name: Alert on production build failure
if: failure()
run: |
curl -X POST -H 'Content-type: application/json' \
--data '{"text":"dbt production build FAILED on main - see run logs"}' \
${{ secrets.SLACK_WEBHOOK_URL }}What Happens to This Pipeline as the Project and Team Both Grow
Everything in this module works well for a project of a few hundred models and a team merging a handful of PRs a day. It is worth understanding, concretely, what starts to strain as both numbers grow significantly, because a system-design interviewer asking about dbt CI/CD often follows up with exactly this kind of scaling question.
| Growth dimension | What starts to strain | The mitigation |
|---|---|---|
| Number of models in the project | Even Slim CI's per-PR cost is bounded by how far downstream a change reaches — a change to a very widely-referenced core model can still trigger a large build. | This is a modeling problem more than a CI problem: minimizing unnecessary fan-out from core shared models (per this track's project-structure module) keeps state:modified+ builds small even as total project size grows. |
| Number of concurrent open PRs | Each open PR holds its own live CI schema and warehouse compute reservation for as long as it stays open, and a warehouse's concurrency limits are finite. | Auto-expiring stale CI schemas (tearing down a PR's schema after some inactivity window, not just on PR close) and a dedicated, appropriately-sized CI warehouse separate from production's warehouse. |
| Number of engineers merging per day | The staging-tier cross-PR-conflict risk from Part 10 grows directly with how many independent changes are in flight at once. | This is exactly why Part 10's staging/QA tier becomes worth its added maintenance cost specifically past a certain team size, even though a smaller team can reasonably skip it. |
| Warehouse compute cost of CI itself | Slim CI is far cheaper than a full build, but it is not free — dozens of daily PRs each building a real, if small, slice of the DAG adds up. | A dedicated, appropriately small CI warehouse size (not the same size provisioned for the full nightly production build) and monitoring CI compute cost as its own line item, not lumped in with production spend. |
The through-line across all four rows: none of these growing pains are fixed by abandoning Slim CI or CI in general — they are fixed by applying the same core principle (build only what is actually needed, scoped as narrowly as correctness allows) one level further, to schema lifecycle, warehouse sizing, and team process, rather than only to model selection.
Handling Seeds, Macro-Only Changes, and Documentation in the Same CI Pipeline
Everything so far has focused on model changes, because they are the most common and highest-risk kind of PR. A mature CI setup also needs to correctly handle the other kinds of changes a dbt project sees regularly — a seed file update, a macro-only change with no model edits, and documentation-only edits — each of which interacts with state:modified+ slightly differently than a plain model change does.
Seed changes — state:modified+ does cover them, but they need dbt seed first
A changed CSV file under seeds/ is picked up by state:modified+ just like a changed model, since seeds are nodes in the same manifest and the same DAG. The detail easy to miss: dbt build does include seeds by default, but a CI pipeline that was written assuming only .sql files ever change (for instance, a workflow trigger path filter that only watches models/**) will silently never fire at all for a seed-only PR.
on:
pull_request:
paths:
- 'models/**'
- 'macros/**'
- 'tests/**'
- 'seeds/**' # <- easy to forget; a seed-only PR needs this to trigger CI at all
- 'snapshots/**' # <- same issue for snapshot-only changes
- 'dbt_project.yml'
- 'packages.yml'Macro-only changes — often invisible to state:modified+ in a way worth knowing explicitly
This is a genuinely sharp edge worth knowing cold for an interview: state:modified detects a change to a macro's own definition, but it does not automatically treat every model thatcalls that macro as modified, the way it does for a direct ref() dependency change. A macro is not itself a node with downstream dependents tracked the same way a model is — so a bug fix inside a widely-used macro can, in some dbt versions and configurations, build a smaller slice of the DAG than the change actually affects, silently under-testing the blast radius of a macro change.
macros/** as deserving a wider build than the default selector would produce on its own — either a full dbt build for macro-only PRs specifically, or, at minimum, explicit awareness during code review of every model known to call the changed macro. Treating a macro change identically to an ordinary model change, and trusting state:modified+ alone to size the build correctly, is a real and easy-to-hit gap.Documentation-only changes — the one case genuinely safe to skip a full build for
A change only to a column or model description: field in a .yml schema file, with no test or config change alongside it, does not affect anything dbt buildactually executes against the warehouse. Many teams add a lightweight, separate CI check for exactly this case — running only dbt parse (which validates that the YAML and Jinja are syntactically sound without touching the warehouse at all) rather than a full build, keeping a documentation PR's feedback loop fast without needing to reason about state selection for it at all.
# For a PR whose diff touches only description: fields in schema.yml files:
dbt parse
# Validates the project compiles and every ref()/source() still resolves,
# with zero warehouse compute cost -- appropriate specifically because
# nothing in a pure documentation change can produce a wrong query result.CI When a Single Project Deploys to More Than One Warehouse Target
A less common but real variant of this problem: some organizations run the same dbt project against two genuinely different warehouse destinations — a common pattern during a warehouse migration (say, Redshift to Snowflake), or when a regulated business unit requires an entirely separate warehouse instance for compliance reasons. CI for this shape needs to validate a change against both targets, not just one, before merge.
Why running Slim CI twice, once per target, is usually simpler than trying to unify it
A tempting but usually wrong instinct is trying to build one clever CI job that somehow validates both targets at once. In practice, the two targets need their own separate production manifests (since they may have drifted independently, especially mid-migration when one target lags the other), their own separate credentials, and potentially their own separate schema-naming scheme. The simpler and more maintainable answer is running the same Slim CI logic twice, as two independent matrix jobs, each comparing against its own target's production state.
jobs:
slim-ci:
runs-on: ubuntu-latest
strategy:
matrix:
target: [snowflake_prod, redshift_prod]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.11'
- run: pip install dbt-core==1.8.0 dbt-snowflake==1.8.0 dbt-redshift==1.8.0
- run: dbt deps
- name: Fetch the manifest for THIS target specifically
run: |
aws s3 cp s3://my-company-dbt-artifacts/${{ matrix.target }}/manifest.json ./prod-manifest/manifest.json
- name: Slim CI build against this target
run: |
dbt build --select state:modified+ --state ./prod-manifest --defer --target ${{ matrix.target }}Each matrix leg reports its own independent pass/fail status on the PR, so a change that happens to be correct on Snowflake but breaks on Redshift (a dialect-specific function that does not exist on one of the two engines, for instance) is caught before merge rather than only surfacing when the lagging target's next production run finally executes the change for the first time.
The multi-target case is also a useful reminder of a theme running through this entire module: every mechanism covered — Slim CI, --defer, the manifest-persistence loop, environment tiering, credential scoping — is composable with the others rather than being a single monolithic "dbt CI setup." A team migrating between warehouses, or serving multiple business units, can layer the matrix strategy above on top of everything from Parts 01 through 14 without redesigning any of it, because each piece solves one specific, narrow problem and none of them assume there is only ever one warehouse target in play.
Six Things People Get Wrong About dbt CI/CD
How Real Teams Actually Run dbt CI/CD
Vimeo — cutting a 45-minute full build down to minutes with Slim CI
Vimeo's analytics engineering team described a familiar growing pain: as their dbt project grew past a few hundred models, a full-build CI check on every PR became slow enough that engineers started batching unrelated changes into fewer, larger PRs just to amortize the CI wait time — the opposite of the small, reviewable changes good engineering practice wants. Adopting Slim CI with state:modified+ against a manifest persisted from the previous production run brought typical PR build times down from tens of minutes to a small fraction of that, because most PRs genuinely only touch a handful of models. The team reported it directly changed engineer behavior back toward smaller, more frequent, easier-to-review PRs, simply because CI was no longer a tax on doing that.
Webflow — treating the production manifest as a first-class deployment artifact
Webflow's data platform team built their CI pipeline around treating manifest.json from every successful production deploy as a versioned artifact in its own right, not an incidental build output — uploaded to cloud storage immediately after every production run, exactly as this module's Part 08 describes. This let them decouple their CI tooling from any one CI vendor: because the state-comparison mechanism only depends on having the right manifest file available, they were able to migrate their pipeline between CI providers without changing any of the underlying Slim CI logic, only the YAML that fetches the file.
Attentive — a required CI check as the actual enforcement mechanism, not a suggestion
Attentive's engineering team found that having a Slim CI workflow existed was not, by itself, enough — until the check was configured as a required status check in their branch protection rules, a determined engineer could and occasionally did merge a PR with a red or still-running CI check during a deadline crunch. Making the Slim CI check a hard merge requirement, not just an informational one, was the specific change that turned CI from "a thing that runs" into "a thing that actually prevents broken models from reaching production" — a distinction the team described as underrated relative to the effort of building the pipeline itself.
A shared lesson across all three: the manifest is a deployment artifact, not an implementation detail
Across each of these three accounts, the same underlying idea recurs in slightly different words: once a team stops treating manifest.json as a throwaway build byproduct and starts treating it as a versioned, deliberately persisted artifact — something published on every successful production run and fetched deliberately at the start of every CI run — Slim CI stops being a fragile, occasionally-broken optimization and becomes a reliable, boring piece of infrastructure nobody has to think about day to day. The engineering effort worth investing here is almost entirely in that persistence-and-retrieval plumbing, not in the dbt command itself, which is a single, unchanging line once the plumbing around it is solid.
7 Questions Interviewers Actually Ask About dbt CI/CD
Six Mistakes Teams Make Rolling Out dbt CI/CD
- ✓Never persisting the production manifest.json anywhere retrievable, so state:modified+ has nothing valid to compare against and CI silently falls back to selecting everything, quietly turning a Slim CI setup back into a full-build setup with none of the intended speed benefit.
- ✓Using hardcoded table references instead of ref()/source() somewhere in the project, creating a gap in the dependency graph that state comparison cannot see through -- a change upstream of that hardcoded reference never triggers a rebuild of what depends on it downstream.
- ✓Making the CI check informational rather than a required, enforced status check in branch protection settings, which means a red or still-running check does not actually block anyone from merging under time pressure.
- ✓Running the exact same full, unfiltered dbt build in both the PR-time CI job and the scheduled production job, missing the entire point of Slim CI -- the two pipelines exist to optimize for different things (review speed versus warehouse-wide consistency) and should usually not use the same selector.
- ✓Forgetting to also publish the freshly generated manifest.json after every production deploy, which silently breaks the loop -- the very next PR's Slim CI run keeps comparing against an increasingly stale reference state instead of the most recent production truth.
- ✓Assuming a macro-only change is fully covered by state:modified+ the same way a model change is, when macro impact on calling models is not always fully tracked the same way a direct ref() dependency is -- treating every macro-touching PR as deserving a wider, more conservative build closes this gap.
Errors You Will Actually Hit Setting This Up
Compilation Error: Model 'X' depends on a node named 'Y' which was not foundWhy: This shows up in a Slim CI run when a model was excluded by state:modified+ (correctly -- it did not change) but --defer was not passed, or was passed without a valid --state path, so dbt has no fallback and tries to find the table in the CI schema where it was never built.
Fix: Add --defer --state <path to the fetched production manifest> to the build command, and confirm the manifest was actually downloaded successfully before the build step runs (a silently failed S3 download is a very common root cause of this specific error).
Database Error: Object 'ANALYTICS.PR_482.STG_ORDERS' does not exist or not authorizedWhy: The CI role/user does not have privileges on the dynamically-created PR-specific schema, usually because the schema-creation grant was set up for one fixed CI schema name rather than the pattern of names Slim CI generates per PR number.
Fix: Grant the CI role privileges on the schema pattern used (e.g. a wildcard grant on schemas prefixed pr_, or a CREATE SCHEMA grant at the database level scoped to the CI role) rather than a single hardcoded schema name.
state:modified+ selects far more models than expected for a small changeWhy: This is usually not a bug -- it means the changed model sits high in the DAG and genuinely has many downstream dependents. It can also mean the comparison manifest is stale (pointing at a much older production state than intended), making dbt see many unrelated accumulated differences as "modified" all at once.
Fix: Confirm the fetched manifest actually corresponds to the most recent production deploy, not an old cached copy; if it does, and the selection is still large, that is correct behavior -- the fix is reconsidering whether that upstream model should have so many direct downstream dependents in the first place.
CI passes locally with dbt build --select state:modified+ but fails only in the pipelineWhy: Almost always an environment difference: the local run has a manifest from a recent local dbt run being compared, while the pipeline is comparing against the real, separately-fetched production manifest -- these are not the same reference file unless deliberately kept in sync.
Fix: Always test locally against the exact same manifest file the CI pipeline actually fetches (download it manually for a local repro) rather than whatever manifest happens to be sitting in your local target/ directory from a previous local invocation.
Two open PRs touching overlapping models silently interfere with each other's CI runsWhy: This happens when the CI schema naming is not actually unique per PR -- for example, a fixed schema name shared across all CI runs rather than one derived from the PR number -- so two concurrent CI builds race to create and drop the same schema.
Fix: Derive the CI target schema name from something guaranteed unique per PR, such as the PR number or the branch name, exactly as shown in this module's GitHub Actions example, so concurrent PRs never share a schema.
A production deploy succeeds and every test passes, but a stakeholder reports a wrong number hours laterWhy: This is not a CI failure at all -- it is a test-coverage gap. CI can only catch what a written test actually checks; a business rule nobody encoded as a singular test can pass silently while producing a plausible-looking but incorrect number.
Fix: Fix the underlying model AND add the specific missing test in the same PR, per this module's Part 12 guidance on the three distinct production-failure shapes -- treat the incident as evidence of exactly which assertion was missing, not just a one-off bug to patch.
🎯 Key Takeaways
- ✓A dbt project deserves the same CI discipline as application code — an automated build and test run on every PR, before merge, not after a scheduled production run finds the problem first.
- ✓Slim CI (state:modified+ against a persisted production manifest.json) makes CI cost scale with the size of a change instead of the size of the whole project, which is the difference between CI being a fast feedback loop and CI being a bottleneck that discourages small PRs.
- ✓--defer and --state work together with state:modified+ so an unbuilt upstream model still resolves correctly, to the real production table, rather than failing because it does not exist in an isolated CI schema.
- ✓PR-time CI and the scheduled production deploy job are two separate pipelines solving two different problems — review-time speed versus warehouse-wide consistency — and conflating them (running the same selector in both) gives up the benefit of each.
- ✓dbt Cloud's built-in CI job and a self-hosted GitHub Actions pipeline both implement the same underlying Slim CI mechanism; the choice between them is a managed-versus-owned trade-off, not a difference in what dbt itself can do.
- ✓A production build failure and a PR-time CI failure are not the same severity — a failed test after a successful production rebuild means wrong data may already be live, which usually calls for an immediate revert rather than a leisurely fix-forward, while an unmerged PR failure carries no such urgency.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.