Databricks and Unity Catalog
Understand Databricks as a cloud-hosted lakehouse platform and Unity Catalog as its centralized governance layer, without tying the concepts to a single provider.
Prerequisites and objective
Know the basics of object storage, tables, SQL, and distributed processing. This lesson focuses on the platform and governance concepts, not on a specific Azure, AWS, or Google Cloud deployment procedure.
You will be able to describe the lakehouse workload model, read a Unity Catalog object hierarchy, write a qualified table name, and explain how identity and storage permissions fit together.
Databricks is a platform, not a cloud provider
Databricks supplies a collaborative data and AI platform that can run with supported cloud providers. Teams use it for notebooks, jobs, SQL analytics, Spark processing, and lakehouse data management. The underlying cloud still supplies networking, object storage, identity, and billing boundaries. Workspace setup and feature availability vary by cloud and account configuration, so verify deployment-specific instructions in current product documentation.
A lakehouse organizes data in open storage formats while adding table management, transactional behavior, and query services. A typical journey lands raw records, validates and transforms them, then publishes tables for analytics. A workspace is where people develop and operate workloads; it is not itself a substitute for an organization-wide data model.
Unity Catalog's governance hierarchy
Unity Catalog organizes securable objects into a hierarchy. Catalogs and schemas help create administrative boundaries, while tables, views, volumes, and functions represent governed assets. Use clear names that signal ownership and data domain rather than making one catalog per notebook.
Metastore
└── Catalog (for example: analytics)
└── Schema (for example: sales)
├── Table (for example: orders)
├── View
└── Function
Workspace users and groups receive privileges on governed objects.
External locations connect governed storage paths to cloud object storage.External locations are governed references to storage paths and their cloud credentials. A principal that can query a table does not automatically need broad direct access to every object in the storage account. Keep the platform permission model and cloud storage permissions aligned, and grant only the needed operations.
Read governed data using SQL
Fully qualified names make ownership and environment boundaries explicit. A data team can publish a stable table for analysts while keeping transformation details in its own schemas.
-- Use a fully qualified name: catalog.schema.table
SELECT region, COUNT(*) AS order_count, SUM(net_amount) AS revenue
FROM analytics.sales.orders
WHERE order_date >= DATE '2026-01-01'
GROUP BY region
ORDER BY revenue DESC;
-- A data owner grants only the required query privilege.
GRANT SELECT ON TABLE analytics.sales.orders TO analyst_group;
-- Review grants and ownership in the platform's governance interface
-- before exposing the table to another team.Build a simple transformation path
The following example rejects unusable records, derives a date, and appends cleaned data to a managed table. Production work should also validate schema changes, deduplicate retries, define a backfill plan, and make writes safe to rerun.
# Example Spark transformation. A Spark session is provided by the job runtime.
from pyspark.sql import functions as F
orders = spark.read.format("delta").table("analytics.bronze.orders")
clean = (
orders
.filter(F.col("order_id").isNotNull())
.filter(F.col("net_amount") >= 0)
.withColumn("order_date", F.to_date("created_at"))
)
(clean.write
.format("delta")
.mode("append")
.saveAsTable("analytics.silver.orders"))Keep source, cleaned, and consumer-facing tables separated by access and lifecycle expectations. A job identity typically needs read access to its inputs and write access only to its output, rather than broad administrator privileges.
Common mistakes
- Calling Unity Catalog cloud-neutral: it is a Databricks governance product, even though its concepts are useful across supported cloud deployments. Separate the general pattern from provider-specific setup.
- Granting workspace admin for convenience: broad roles make changes hard to audit. Use groups, data ownership, and narrow privileges.
- Confusing a catalog with a storage bucket: a catalog is a governed namespace, not necessarily a physical directory. Map it to data ownership intentionally.
- Ignoring cloud permissions: catalog-level permissions alone may not grant the underlying storage access. Review both authorization layers.
Practice
- Design catalogs and schemas for a retailer's raw sales, cleaned orders, and finance reporting domains.
- Write a fully qualified query for a table named daily_revenue in a finance schema.
- Describe the smallest permissions a scheduled transformation needs on its input and output.
- Draw how a user's identity, a catalog privilege, an external location, and object storage relate.
Hint: Separate who may discover/query an asset from which runtime identity can read or write its backing objects.
🎯 Key Takeaways
- ✓Databricks provides collaborative lakehouse workloads on supported cloud infrastructure.
- ✓Unity Catalog organizes governed assets with a metastore, catalogs, schemas, and securable objects.
- ✓Use qualified names, clear ownership, groups, and least-privilege grants.
- ✓Catalog authorization and cloud storage authorization must both be considered.
- ✓Treat concepts as portable, but confirm provider-specific features and setup in current documentation.
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
