Secrets Management — Vault and Cloud Secret Stores

Keep credentials out of code and images with environment variables, secret managers, access policies, rotation, and safe operational workflows.

Est. 30 min Updated Suggest a fix
// Part 01 — The problem

A Secret Is a Credential with a Lifecycle

A delivery API needs a database password, a payment integration token, and a signing key. Each value grants access to something valuable, so it must be protected from source control, build logs, container images, untrusted users, and unnecessary copies. Hiding a string in a different file is not enough: a secret needs an owner, a reason to exist, an access policy, a rotation plan, and a retirement step.

Secret management is the process of creating, storing, granting, delivering, auditing, rotating, and revoking sensitive values. A password in a managed secret store still needs least-privilege access. A secret injected into a process can still leak through debug output, crash reports, or a compromised host. Design the full lifecycle, not only the storage location.

ValueExampleFirst question
CredentialDatabase password or API tokenWhich identity needs it, and for which operation?
Private keySigning or client-authentication keyCan the platform keep it in a managed key service instead?
ConfigurationPublic API base URL or log levelIs it actually confidential, or just environment-specific?
Personal dataCustomer address or payment detailIs it being incorrectly treated as application configuration?
💡 Note
Not every configuration value is a secret. Classify data deliberately: encrypting and restricting all settings can make operations harder, while treating credentials as ordinary config creates avoidable exposure.
// Part 02 — Local development

Use Environment Variables Without Committing Values

Environment variables are a convenient interface for local tools and processes, but they are not a secret store by themselves. A local .env file can help a developer run an app if it is excluded from version control and protected by the workstation. Do not commit populated files, paste credentials into shell history, or assume a variable is hidden from every process on the same host.

.env.example
bash
DATABASE_URL=
PAYMENTS_API_TOKEN=

Commit a blank template that documents required names, and keep actual values in a protected local file or approved developer secret tool. Add the real filename to the repository ignore rules before placing credentials in it. The application should fail clearly when a required value is missing; it should not silently connect to a production service with an empty or fallback credential.

settings.py
python
import os


def required_secret(name: str) -> str:
    value = os.environ.get(name)
    if not value:
        raise RuntimeError(f"Required secret {name} is not configured")
    return value


def load_settings() -> dict[str, str]:
    return {
        "database_url": required_secret("DATABASE_URL"),
        "payments_api_token": required_secret("PAYMENTS_API_TOKEN"),
    }


if __name__ == "__main__":
    settings = load_settings()
    print("Required settings are present; values were not printed.")
⚠️ Important
Never print a settings object containing credentials to debug a deployment. Logs, screenshots, CI output, and support tickets can outlive the terminal session and have a much wider audience.
// Part 03 — Choose a store

Use a Secret Manager for Shared and Production Workloads

A cloud secret manager or HashiCorp Vault provides controlled storage, identity-based access, audit events, and often versioning or rotation integrations. Exact features and policy syntax vary by provider and deployment; follow the official documentation for the store your organization operates. Keep the application contract small: the workload authenticates using its assigned identity, requests only the named secret it needs, and receives the value at runtime.

ApproachUseful forImportant limitation
Developer environment variableLocal run or a small process boundaryCan be exposed to local processes, shell tooling, or diagnostics
CI/CD protected secretA deployment job that needs a narrowly scoped credentialForks, pull requests, logs, and job permissions need careful controls
Cloud secret managerProduction workload needing central policy and auditThe workload identity and network path still need protection
HashiCorp VaultOrganizations requiring centralized dynamic or leased credentialsOperators must configure, secure, upgrade, and recover the Vault service
Kubernetes SecretPassing a value through the Kubernetes API to a workloadBase64 representation is not encryption; cluster access and at-rest controls matter

Prefer workload identity or short-lived credentials over a long-lived access key stored in an image or manifest. Scope access to the exact secret and environment, separate staging from production, and audit reads. If a cloud provider or Vault is unavailable, the application should fail safely and surface a useful operational error without logging the requested value.

// Part 04 — Access design

Grant Access to Workloads, Not to Every Engineer

Start from the action: which service needs which credential, in which environment, and for how long? Give that service identity read access to a specific secret rather than broad access to an entire account or namespace. Human access should be individually attributable and time-bounded where the platform supports it. Remove permissions when a workload or person no longer needs them.

  1. Name the workload identity and its owner; do not share one administrator credential across applications.
  2. Grant only the read or use operation required on the specific secret and environment.
  3. Keep production access separate from development and staging access.
  4. Enable audit events and alert on unexpected access patterns without recording secret values.
  5. Review access when owners, deployment paths, or application dependencies change.

A secret store is one layer in the access design, not a way to repair a compromised application. If a web handler can read the payment token, an attacker who takes over that handler may be able to use it. Limit what the token itself can do, use provider-side restrictions where available, and monitor the operations performed with it.

🎯 Pro Tip
Store a non-secret identifier such as PAYMENTS_TOKEN_SECRET_NAME in deployment configuration, then let the workload identity retrieve that named secret. Avoid turning the identifier into an excuse for broad store permissions.
// Part 05 — Delivery

Inject Secrets at Runtime, Not While Building the Image

A container image is a reusable artifact that may be pushed to a registry, cached, scanned, and shared across environments. Do not bake environment credentials into a Dockerfile, build argument, source layer, or image filesystem. The deployment platform should provide the runtime value through an approved secret integration, a mounted file, or another controlled mechanism that fits the application.

deployment-secret-reference.yaml
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      serviceAccountName: orders-api
      containers:
        - name: api
          image: ghcr.io/example-org/orders-api:replace-with-commit-sha
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: orders-api-runtime
                  key: DATABASE_URL

This manifest references a Kubernetes Secret object; it does not fetch a cloud secret by itself. In a real cluster, create or synchronize that object through an approved secret manager integration, enable the cluster's appropriate encryption and access controls, and limit which service accounts can read it. Never commit the secret value in the manifest. The image tag is illustrative: use an immutable, reviewed release artifact.

⚠️ Important
A secret mounted as an environment variable or file may still be visible to the application, diagnostics, or users who can inspect the process or pod. Restrict workload inspection access and choose the delivery mode with those boundaries in mind.
// Part 06 — Rotation and response

Plan Rotation Before a Credential Is Leaked

Rotation replaces a credential with a new one and removes the old one after dependent workloads have moved over. Some backends support overlapping credentials: create a replacement, update consumers, verify success, then revoke the old value. Other systems have a short interruption or require coordinated changes. Test the actual dependency and define a rollback that does not re-enable a compromised credential.

  1. Identify every consumer and owner of the credential before changing it.
  2. Create or request a replacement using the approved provider workflow.
  3. Update the secret store and roll out consumers using a controlled deployment.
  4. Verify authentication and business health, then revoke the previous credential.
  5. Record completion and update the next rotation or review checkpoint.

If exposure is suspected, revoke or disable the credential at its issuer first, then investigate where it appeared, what it could access, and which actions used it. Rotate dependent credentials if they were reachable from the compromised context. Remove the value from exposed locations where possible, but do not assume deleting a Git commit or log entry makes a leaked credential safe again.

📌 Real World Example
For a database used by a busy order service, coordinate a credential change with connection pools and background workers. Confirm the new credential works on all replicas before removing the old one, and watch authentication failures during the rollout.
// Part 07 — Delivery pipelines

Protect Secrets in CI and Deployment Workflows

A CI job should receive only the credentials required for that job and only from trusted workflow contexts. A pull request from an external contributor must not be able to run arbitrary code with production credentials. Prefer short-lived deployment identity where supported; otherwise scope a protected secret to the smallest environment and require review for the deployment stage.

Masking a value in logs is a useful safeguard, not a guarantee: transformed or encoded values may not be redacted. Avoid commands that echo secrets, do not pass them as command-line arguments when they may appear in process listings, and never put credentials in artifact contents. Keep deployment logs useful by reporting identity, environment, version, and outcome without printing the secret itself.

Pipeline boundarySafer practiceReview question
Pull request testsRun without production credentialsCan untrusted code reach a privileged job?
Build imageBuild from source without runtime credentialsCould a layer or build log retain a secret?
Deploy to stagingUse staging-only identity and secretsDoes the job have access beyond this environment?
Production releaseRequire approved, traceable promotionWho authorized the change and which artifact was deployed?
// Part 08 — Pitfalls

Common Mistakes

Committing a populated .env file

Why: Local convenience is mistaken for repository privacy, and clones or history retain the value.

Fix: Commit a blank example only, ignore the real file, and rotate any credential that was exposed.

Embedding a credential in a container image

Why: Image layers and registries preserve build-time contents across deployments.

Fix: Inject a runtime secret from an approved store and rebuild clean images after exposure.

Giving every workload broad secret-store access

Why: One identity is reused for ease of setup.

Fix: Use separate workload identities and grant per-secret, per-environment permissions.

Treating Kubernetes Secret data as encrypted automatically

Why: Base64 encoding looks opaque but does not provide confidentiality.

Fix: Configure and verify cluster encryption, RBAC, and an approved external secrets workflow.

// Part 09 — Practice

Design a Secret Lifecycle for an Orders Service

Practice

Remove static credentials from a deployment path

  1. List the database and payment credentials used by a sample orders API; identify their owners, scopes, and consumers.
  2. Create a blank environment template and an application settings loader that fails when a required value is absent without printing values.
  3. Draw a production access path from workload identity to one named secret, including the audit and network boundaries.
  4. Write a rotation checklist that covers a replacement, rollout verification, old credential revocation, and failure rollback.
  5. Review a CI workflow design and mark which jobs can run untrusted code and which jobs may access deployment credentials.

Hint: Keep every exercise credential fictional. The goal is to reason about scope and lifecycle, not to create a real cloud key or store it in the repository.

🎯 Key Takeaways

  • ✓A secret has a lifecycle: create, scope, deliver, audit, rotate, and revoke it.
  • ✓Environment variables are a delivery interface, not a secret manager; protect local files and avoid exposing values in logs.
  • ✓Use workload identity and least privilege so each service can retrieve only the credentials it needs.
  • ✓Do not bake runtime credentials into source code, container images, build logs, or deployment manifests.
  • ✓Kubernetes Secret objects require appropriate encryption, access controls, and an approved provisioning path.
  • ✓Plan rotation in advance and revoke exposed credentials at their issuer; deleting a copy does not make it safe again.

Next

What comes next

You have designed how applications receive sensitive values. Next, build a delivery pipeline that tests, packages, and deploys changes without leaking those credentials.

Lesson 10 · CI/CD with GitHub Actions →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...