Azure Key Vault for Data Engineers — Stop Putting Secrets in Your Code
Secrets in code are the most common security mistake in data engineering. Connection strings, API keys, and storage account keys hardcoded in notebooks, pipeline configs, or environment variables get committed to Git and end up exposed. Azure Key Vault is the solution — and it takes 15 minutes to set up properly.
What Key Vault does
Azure Key Vault is a managed secrets store. You store secrets (passwords, connection strings, API keys, certificates) in Key Vault and retrieve them at runtime using managed identity — no credentials in your code at all.
Audit logs track every secret access. You can rotate secrets without changing any code. You can revoke access to a specific secret immediately without touching other resources.
Key Vault with Azure Databricks
In Databricks, create a secret scope backed by Key Vault:
dbutils.secrets.get(scope="kv-scope", key="storage-account-key")
Databricks calls Key Vault at runtime using its Managed Identity. The secret value never appears in notebook output or logs. Even if someone shares the notebook, they cannot see the actual secret value.
This is the correct way to store ADLS access keys, database connection strings, and API keys used in Databricks notebooks.
Key Vault with Azure Data Factory
In ADF, create a Linked Service and select Azure Key Vault as the authentication method. ADF retrieves the secret at pipeline runtime using its Managed Identity.
Best practice: store all ADF Linked Service passwords in Key Vault. This means your ADF pipeline JSON configuration contains only Key Vault references — no actual credentials. Safe to commit to Git, safe to share with colleagues.
Setting it up in 15 minutes
1. Create a Key Vault in Azure Portal
2. Add your secrets (storage account key, database password, API keys)
3. Grant your Databricks Managed Identity the Key Vault Secrets User role
4. In Databricks: create a secret scope pointing to your Key Vault URL
5. Replace all hardcoded secrets with dbutils.secrets.get() calls
For ADF: enable System Managed Identity on the ADF instance, grant it Key Vault Secrets User, and update Linked Services to reference Key Vault secrets.