Setting Up a dbt Project

dbt Core vs dbt Cloud — Same Language, Different Runtime

Before touching a single file, you need to decide where dbt actually runs. dbt is a transformation tool, not a database — it compiles SQL and Jinja into plain SQL and sends it to your warehouse to execute. But the thing doing that compiling and sending has to live somewhere, and dbt gives you two genuinely different options: dbt Core and dbt Cloud. They share the same modeling language and the same project files, so nothing you learn in one is wasted if you switch to the other later.

dbt Core is the open-source command-line tool. You install it yourself (typically via pip), you run it from your own machine or your own CI/CD system, and you are responsible for scheduling it — cron, Airflow, GitHub Actions, Dagster, whatever your team already uses to run jobs on a schedule. There is no hosted UI. Everything happens through thedbt CLI and whatever code editor you already use.

dbt Cloud is a managed product built on top of dbt Core. It gives you a browser-based IDE for writing and testing models, a built-in job scheduler so you don't need a separate orchestration tool just to run dbt, hosted documentation, and a metadata API for other tools to query your project's state. Under the hood, dbt Cloud is still running the same dbt Core engine — it is not a different modeling language, it is a different place for that engine to execute and a set of operational conveniences wrapped around it.

Installing dbt Core — One Package Per Warehouse

dbt does not ship with support for every warehouse baked into one giant package. Instead, it uses an adapter pattern: a small, warehouse-specific package translates dbt's general instructions into the SQL dialect and connection protocol that specific warehouse understands. You install dbt-core plus exactly one adapter package for the warehouse you actually use.

# Create an isolated Python environment first — dbt has its own
# dependency versions and you don't want them colliding with other
# Python projects on the same machine
python3 -m venv dbt-env
source dbt-env/bin/activate        # on Windows: dbt-env\Scripts\activate

# Install dbt-core AND the adapter for your warehouse together.
# Installing the adapter automatically pulls in dbt-core as a dependency —
# you do not need to install dbt-core separately.
pip install dbt-snowflake

# Verify the install and see which adapter dbt picked up
dbt --version

Core:
  - installed: 1.8.3
  - latest:    1.8.3 - up to date!

Plugins:
  - snowflake: 1.8.2 - up to date!

If your warehouse is different, the package name changes but the pattern doesn't. This is the single most common first mistake people make when installing dbt — they run pip install dbt-core alone, get a working CLI, and then get a confusing error the moment they try to connect to a warehouse, because no adapter was ever installed to talk to it.

dbt_project.yml — The File That Makes a Folder a dbt Project

A dbt project is, at minimum, a folder containing one specific file: dbt_project.yml, sitting at the project's root. This file is how dbt recognizes "this directory is a dbt project" — when you run any dbt command, dbt walks upward from your current directory looking for this file to figure out where the project root is. It defines the project's name, which dbt version it expects, where to look for models and other resources, and default configuration that applies across the whole project unless a more specific config overrides it.

name: 'freshcart_analytics'
version: '1.0.0'
config-version: 2

# Which adapter-specific profile (from profiles.yml) this project uses
profile: 'freshcart'

# Where dbt looks for each type of resource, relative to the project root
model-paths: ['models']
seed-paths: ['seeds']
test-paths: ['tests']
macro-paths: ['macros']
snapshot-paths: ['snapshots']
analysis-paths: ['analyses']

# Directories dbt is allowed to delete when you run `dbt clean`
clean-targets:
  - 'target'
  - 'dbt_packages'

# Default materialization settings, applied per directory inside models/
# These are defaults — any individual model can override them with its
# own config({...}) block, which always wins over what's set here.
models:
  freshcart_analytics:
    staging:
      +materialized: view
    marts:
      +materialized: table
      finance:
        +materialized: table
        +tags: ['finance']

Two fields deserve extra attention because they trip people up constantly. name is your project's internal identifier — it is also the top-level key you use inside themodels: block to scope configuration to this project (notice freshcart_analytics appears both at the top and nested under models: — that is not a coincidence, it has to match). profile is the name dbt looks up inside your separate profiles.yml file to find actual connection credentials — it is a pointer, not the credentials themselves.

The models: block is where per-directory materialization defaults live. Here, anything undermodels/staging/ defaults to a view, and anything under models/marts/ defaults to atable, with a further override formodels/marts/finance/ that also tags those models. The + prefix on each config key is dbt's YAML convention for "this is a config setting being applied to everything at and below this path," not a literal part of the setting's name.

profiles.yml — Credentials Live Outside the Project, On Purpose

dbt_project.yml defines what your project looks like. It never contains a password, an account name, or a warehouse identifier. Those live in a completely separate file called profiles.yml, and critically, that file lives outside your project directory entirely — by default at ~/.dbt/profiles.yml, in your home directory. This separation is deliberate: your dbt project gets committed to git and shared with your team, but your personal warehouse credentials should never end up in version control.

freshcart:                              # matches `profile: 'freshcart'` in dbt_project.yml
  target: dev                           # which target below is used by default

  outputs:
    dev:
      type: snowflake
      account: fc12345.us-east-1
      user: asil_dev
      password: "{{ env_var('DBT_SNOWFLAKE_PASSWORD') }}"
      role: TRANSFORMER_DEV
      database: FRESHCART_DEV
      warehouse: TRANSFORMING_XS
      schema: dbt_asil                  # each developer gets their own schema
      threads: 4

    prod:
      type: snowflake
      account: fc12345.us-east-1
      user: svc_dbt_prod
      private_key_path: "{{ env_var('DBT_SNOWFLAKE_KEY_PATH') }}"
      role: TRANSFORMER_PROD
      database: FRESHCART_PROD
      warehouse: TRANSFORMING_L
      schema: analytics
      threads: 8

Notice the two targets, dev and prod, authenticate differently. The dev target uses a password pulled from an environment variable viaenv_var() — never hardcoded in the file itself, even though the file lives outside git's reach anyway; defense in depth matters. The prod target uses a private key file instead of a password, which is the more common pattern for a service account that runs scheduled production jobs rather than a human logging in interactively. Snowflake supports both, and which one you choose is a security and operations decision, not a dbt one.

The schema field is worth calling out specifically: giving each developer their own schema (dbt_asil,dbt_maria, and so on) via the dev target means everyone can run dbt run against their own isolated copy of the models without stepping on each other's tables, while prod writes to the single sharedanalytics schema that actual dashboards query against.

The Standard Folder Structure dbt Expects

A dbt project has a small number of top-level directories, each with one clear job. You don't have to use all of them from day one, but every serious dbt project eventually does, and knowing what belongs where prevents a project from turning into a folder of miscellaneous SQL files with no organizing logic.

freshcart_analytics/
├── dbt_project.yml
├── packages.yml                  # third-party dbt packages this project depends on
├── models/                       # SELECT statements — the transformations themselves
│   ├── staging/
│   │   └── stg_orders.sql
│   ├── intermediate/
│   │   └── int_orders_joined.sql
│   └── marts/
│       └── fct_orders.sql
├── tests/                        # custom, singular data tests (not schema tests)
│   └── assert_positive_order_totals.sql
├── macros/                       # reusable Jinja functions, callable from any model
│   └── cents_to_dollars.sql
├── seeds/                        # small, static CSV files dbt loads as tables
│   └── country_codes.csv
├── snapshots/                    # slowly changing dimension history-tracking
│   └── snapshot_customers.sql
├── analyses/                     # ad-hoc SQL that dbt compiles but never runs
│   └── revenue_by_region_adhoc.sql
└── target/                       # dbt's generated output — never hand-edited, never committed

The distinction between analyses/ andmodels/ is one people frequently miss: a file inanalyses/ gets the full benefit of Jinja compilation — you can use ref(), source(), and macros inside it — but dbt never creates a table or view from it. It exists purely so you can write and version-control one-off exploratory or reporting queries using the same building blocks as your real models, without those queries becoming part of your actual DAG.

The CLI Commands You Need Before You Ever Model Anything

Most dbt tutorials jump straight to dbt run. But before you ever get a model to run successfully, a handful of other commands do the actual diagnostic and setup work — and you'll return to them constantly throughout the life of a real project, not just on day one.

dbt debug — is the connection even working?

dbt debug is the very first command to run in any new or unfamiliar project. It does not touch your models at all. It checks that dbt_project.yml is valid, that a matching profile exists in profiles.yml, and — most importantly — that dbt can actually open a connection to your warehouse using those credentials.

$ dbt debug

dbt version: 1.8.3
python version: 3.11.6
python path: /Users/asil/dbt-env/bin/python3
os info: macOS-14.5-arm64

Using profiles.yml file at /Users/asil/.dbt/profiles.yml
Using dbt_project.yml file at /Users/asil/freshcart_analytics/dbt_project.yml

Configuration:
  profiles.yml file [OK found and valid]
  dbt_project.yml file [OK found and valid]

Required dependencies:
 - git [OK found]

Connection:
  account: fc12345.us-east-1
  user: asil_dev
  database: FRESHCART_DEV
  warehouse: TRANSFORMING_XS
  role: TRANSFORMER_DEV
  schema: dbt_asil
  Connection test: [OK connection ok]

All checks passed!

When a check fails, dbt debug tells you exactly which one — an invalid account identifier, a wrong password, a role that doesn't have USAGE on the target warehouse — instead of you discovering the same problem twenty minutes later as a cryptic error buried inside a full model run. Run it any time a new teammate sets up the project locally, any time credentials rotate, or any time a CI job starts failing for no obvious reason.

dbt deps — installing package dependencies

Many dbt projects depend on community packages — most commonlydbt_utils for generic helper macros. Package dependencies are declared in a separate packages.ymlfile at the project root, and dbt deps downloads them into the dbt_packages/ directory.

# packages.yml — at the project root, alongside dbt_project.yml
packages:
  - package: dbt-labs/dbt_utils
    version: [">=1.1.0", "<2.0.0"]

$ dbt deps

Installing dbt-labs/dbt_utils
  Installed from version 1.1.1
  Up to date!

Installed 1 package(s) in 0.87s

dbt build — run, test, seed, and snapshot together, in DAG order

dbt run only runs models. It does not run your tests, load your seeds, or apply your snapshots. In a real deployment you almost always want all four to happen together, and in the correct dependency order — a seed a model depends on loaded before that model runs, and tests for a model run immediately after that specific model builds, rather than only at the very end. dbt build does exactly this: it runs seeds, snapshots, models, and tests as one unified DAG.

# Only runs models — seeds, snapshots, and tests are untouched
dbt run

# Runs seeds, snapshots, models, AND tests, interleaved by DAG dependency —
# a test on stg_orders runs right after stg_orders builds, not after
# the entire project finishes
dbt build

The practical benefit of dbt build's interleaving is fail-fast behavior: if a test on an early staging model fails,dbt build stops that model's downstream dependents from building on top of bad data, rather than running your entire project first and only discovering the problem in a separate test pass at the very end. Most production dbt jobs usedbt build, not dbt run, for exactly this reason.

dbt clean — clearing generated and downloaded artifacts

dbt clean deletes the directories listed underclean-targets in dbt_project.yml — typically target/ and dbt_packages/. It's a blunt reset button: useful when a stale compiled artifact or a corrupted package install is causing confusing behavior, and you want to force everything to regenerate from scratch on the next command.

A Full Walkthrough — dbt init to Your First Successful dbt run

Everything so far has been individual pieces. Here is the whole sequence, start to finish, for standing up a brand-new dbt project against a real Snowflake warehouse and getting a model to actually build.

$ dbt init freshcart_analytics

Running with dbt=1.8.3

Which database would you like to use?
[1] snowflake

Enter a number: 1

account (https://<this_value>.snowflakecomputing.com): fc12345.us-east-1
user (dev username): asil_dev
[1] password
[2] keypair
[3] sso
Desired authentication type option (enter a number): 1
password (dev password): ********
role (dev role): TRANSFORMER_DEV
warehouse (dev warehouse): TRANSFORMING_XS
database (dev database): FRESHCART_DEV
schema (dev schema): dbt_asil
threads (1 or more): 4

Profile freshcart_analytics written to /Users/asil/.dbt/profiles.yml
using target's profile_template.yml and your supplied values. Run 'dbt
debug' to validate the connection.

Your new dbt project "freshcart_analytics" was created!

dbt init does two things at once: it generates the standard project skeleton (Part 05's folder structure, plus a starter dbt_project.yml) in a new directory named after your project, and it interactively prompts for connection details, writing the result to ~/.dbt/profiles.ymlfor you — you never have to hand-write your firstprofiles.yml from Part 04's template, though you will often go back and add a second target (like prod) by hand afterward.

$ cd freshcart_analytics
$ dbt debug

Connection test: [OK connection ok]
All checks passed!

$ dbt run

Running with dbt=1.8.3
Found 2 models, 0 tests, 0 sources, 0 exposures, 0 metrics

Concurrency: 4 threads (target='dev')

1 of 2 START sql view model dbt_asil.my_first_dbt_model ... [RUN]
1 of 2 OK created sql view model dbt_asil.my_first_dbt_model ... [SUCCESS 1 in 1.24s]
2 of 2 START sql view model dbt_asil.my_second_dbt_model .. [RUN]
2 of 2 OK created sql view model dbt_asil.my_second_dbt_model .. [SUCCESS 1 in 0.98s]

Finished running 2 view models in 0 hours 0 minutes and 3.41 seconds (3.41s).

Completed successfully

Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

That first dbt run is the moment everything from Parts 01 through 06 comes together: dbt readsdbt_project.yml to find your models directory, readsprofiles.yml to know which warehouse and schema to write to, compiles the two example .sql files dbt init generated for you into real CREATE VIEWstatements, and executes them againstFRESHCART_DEV.dbt_asil. If this succeeds, your environment is fully wired up and you are ready to delete the example models and start writing real ones — which is exactly where Module 04 picks up.

Switching Between dev, CI, and prod Without Editing Files

A profile can define any number of named targets — Part 04's example had dev and prod, but real projects often add a third, such as ci, pointing at a disposable schema used only by automated pull-request checks. You switch between them with the --target flag rather than editing profiles.yml every time.

# Uses whichever target is listed as the default (target: dev) in profiles.yml
dbt run

# Explicitly overrides which target to use for this invocation only
dbt run --target prod

# In CI, environment variables typically supply a ci-specific target
dbt build --target ci

This is why profiles.yml's structure — one profile name, many named targets underneath it — matters so much in practice: your models, tests, and macros never referencedev or prod directly. They just sayref('stg_orders'), and dbt resolves that to whichever database and schema the active target points at. The exact same project, unmodified, safely builds into a developer's personal sandbox, a CI throwaway schema, or the shared production schema, purely based on which target is active when the command runs.

Running dbt in CI Without a Human Ever Typing a Password

Part 04's profiles.yml examples used{{ env_var('DBT_SNOWFLAKE_PASSWORD') }} rather than a literal password string. That pattern is what makes it possible to run dbt inside an automated pipeline at all — a CI runner has no interactive human to type a password into a prompt, and it should not have a plaintext credential sitting in a file it checks out from git. Environment variables are the bridge between "a secret CI needs" and "a value dbt can read."

freshcart:
  target: "{{ env_var('DBT_TARGET', 'dev') }}"
  outputs:
    dev:
      type: snowflake
      account: "{{ env_var('DBT_SNOWFLAKE_ACCOUNT') }}"
      user: "{{ env_var('DBT_SNOWFLAKE_USER') }}"
      password: "{{ env_var('DBT_SNOWFLAKE_PASSWORD') }}"
      role: "{{ env_var('DBT_SNOWFLAKE_ROLE') }}"
      database: "{{ env_var('DBT_SNOWFLAKE_DATABASE') }}"
      warehouse: "{{ env_var('DBT_SNOWFLAKE_WAREHOUSE') }}"
      schema: "{{ env_var('DBT_SNOWFLAKE_SCHEMA') }}"
      threads: 4

Notice this file itself contains zero secrets — every sensitive value is a reference to an environment variable that must exist wherever dbt runs. A developer's laptop sets these in a local shell profile or a .env file that is itself gitignored; a CI runner sets them as encrypted repository secrets that get injected into the job's environment right before the job runs, and never appear in logs.

# .github/workflows/dbt_ci.yml
name: dbt CI

on: pull_request

jobs:
  dbt-build:
    runs-on: ubuntu-latest
    env:
      DBT_TARGET: ci
      DBT_SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
      DBT_SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_CI_USER }}
      DBT_SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_CI_PASSWORD }}
      DBT_SNOWFLAKE_ROLE: TRANSFORMER_CI
      DBT_SNOWFLAKE_DATABASE: FRESHCART_CI
      DBT_SNOWFLAKE_WAREHOUSE: TRANSFORMING_XS
      DBT_SNOWFLAKE_SCHEMA: ci_pr_${{ github.event.pull_request.number }}
    steps:
      - uses: actions/checkout@v4
      - run: pip install dbt-snowflake==1.8.2
      - run: dbt deps
      - run: dbt build

One detail worth calling out: the schema is set toci_pr_{pull request number}, giving every open pull request its own disposable schema to build into. This means two pull requests running CI at the same time never collide by writing to the same tables, and cleaning up old CI schemas becomes a simple, periodic job rather than a source of data corruption between concurrent runs.

A Pre-Flight Checklist for a New or Inherited dbt Project

Between the pieces covered so far — installation, project files, folder structure, credentials, and the CLI — it's easy to miss a step when either standing up a brand-new project or picking up an existing one you didn't build. Here is the order that catches the most common setup problems fastest, each step building on exactly what earlier Parts of this module covered.

# 1. Confirm the right adapter is installed for this warehouse (Part 02)
dbt --version
# check the "Plugins" section actually lists the expected adapter,
# e.g. snowflake, not just "Core" with no plugin listed

# 2. Confirm dbt_project.yml and profiles.yml agree, and the
#    connection genuinely works (Part 03, Part 04, Part 06)
dbt debug

# 3. Install any declared package dependencies (Part 06)
dbt deps

# 4. Compile without touching the warehouse, to catch Jinja/ref errors
#    early and cheaply, before spending any real compute (Part 06)
dbt compile

# 5. Only now, run the full project for real
dbt build

The reason to insert dbt compile before the first real dbt build, rather than jumping straight to it, is cost and diagnostic clarity: a Jinja typo, a missingref() target, or a broken macro call shows up immediately in a fast, free compile step, rather than only surfacing after a slow, potentially expensivedbt build has already started executing SQL against real warehouse compute.

Five Misconceptions About Setting Up a dbt Project

What This Looks Like on Day One

5 Interview Questions — With Complete Answers

The Setup Mistakes That Waste the Most Time

Errors You Will Hit — And Exactly Why They Happen