Secrets Management in Kubernetes with HashiCorp Vault

The Kubernetes secrets problem

Kubernetes Secrets are base64-encoded, not encrypted. In a default cluster, they're stored in etcd in plaintext, accessible to anyone with cluster-admin permissions, and easily leaked through kubectl get secret -o yaml. For any team handling credentials, API keys, or database passwords, this isn't acceptable.

The three common solutions are: Sealed Secrets (encrypt secrets before committing to Git), External Secrets Operator (pull from AWS Secrets Manager or GCP Secret Manager), and HashiCorp Vault (a dedicated secrets management platform with dynamic secrets and fine-grained access control).

Vault is the most powerful option — and the most complex. This guide covers how to integrate it with Kubernetes in a way that's actually maintainable.

Why Vault over External Secrets + AWS Secrets Manager?

For many teams, the External Secrets Operator pulling from AWS Secrets Manager is the right answer — simpler to operate, good enough security, integrates with existing AWS IAM. You don't need Vault if:

• All your workloads run on AWS
• You don't need dynamic secrets (auto-rotating database credentials)
• You don't have multi-cloud or on-premise requirements

Vault wins when you need:

• Dynamic secrets: Vault generates short-lived database credentials on demand and revokes them when the lease expires. No long-lived passwords.
• Multi-cloud / multi-environment: One Vault cluster can serve AWS, Azure, on-prem, and local development
• Fine-grained audit logs: Every secret access is logged with the identity, time, and exact secret accessed
• Secret leasing: Secrets have TTLs and are automatically revoked

Deploy Vault on Kubernetes

The official Vault Helm chart is the standard deployment approach:

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# vault-values.yaml
server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }
        storage "raft" {
          path = "/vault/data"
        }
        service_registration "kubernetes" {}

  dataStorage:
    enabled: true
    size: 10Gi
    storageClass: gp3

  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
      - host: vault.internal.yourdomain.com
        paths: [/]

ui:
  enabled: true
  serviceType: ClusterIP

helm upgrade --install vault hashicorp/vault   --namespace vault --create-namespace   -f vault-values.yaml

Initialize and unseal

On first run, initialize the cluster to get unseal keys and a root token:

kubectl exec -n vault vault-0 -- vault operator init   -key-shares=5   -key-threshold=3   -format=json > vault-init.json

# Store these keys in a secure location — AWS Secrets Manager, not in Git
# Unseal with 3 of the 5 keys
kubectl exec -n vault vault-0 -- vault operator unseal <key1>
kubectl exec -n vault vault-0 -- vault operator unseal <key2>
kubectl exec -n vault vault-0 -- vault operator unseal <key3>

For production, use auto-unseal with AWS KMS:

# Add to vault-values.yaml server.ha.raft.config
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal"
}

This eliminates manual unseal steps after restarts.

Configure Kubernetes authentication

Vault's Kubernetes auth method lets pods authenticate using their ServiceAccount tokens. Vault validates the token with the Kubernetes API and grants a Vault token in return.

# Enable the Kubernetes auth method
vault auth enable kubernetes

# Configure it with the cluster's API server address
vault write auth/kubernetes/config   kubernetes_host="https://kubernetes.default.svc"   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Create a policy for your application:

# my-app-policy.hcl
path "secret/data/my-app/*" {
  capabilities = ["read"]
}

path "database/creds/my-app-role" {
  capabilities = ["read"]
}

vault policy write my-app my-app-policy.hcl

Bind the policy to a Kubernetes ServiceAccount:

vault write auth/kubernetes/role/my-app   bound_service_account_names=my-app   bound_service_account_namespaces=my-app   policies=my-app   ttl=1h

Inject secrets with the Vault Agent Injector

The Vault Agent Injector is a mutating webhook that automatically injects a Vault Agent sidecar into pods with specific annotations. The agent authenticates to Vault and writes secrets to a shared in-memory volume.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-app
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "my-app"
        vault.hashicorp.com/agent-inject-secret-config.env: "secret/data/my-app/config"
        vault.hashicorp.com/agent-inject-template-config.env: |
          {{- with secret "secret/data/my-app/config" -}}
          export DATABASE_URL="{{ .Data.data.database_url }}"
          export API_KEY="{{ .Data.data.api_key }}"
          {{- end }}
    spec:
      serviceAccountName: my-app
      containers:
        - name: my-app
          image: my-app:latest
          command: ["/bin/sh", "-c", "source /vault/secrets/config.env && exec my-app"]

The agent injects a sidecar that authenticates using the pod's ServiceAccount, fetches the secret, renders it using the template, and writes it to /vault/secrets/config.env. Your application sources the file at startup — no Vault SDK required, no secrets as environment variables in the pod spec.

Dynamic database credentials

This is Vault's killer feature. Instead of a long-lived database password, Vault generates unique credentials for each pod with a 1-hour TTL:

# Enable the database secrets engine
vault secrets enable database

# Configure a PostgreSQL connection
vault write database/config/my-postgres   plugin_name=postgresql-database-plugin   connection_url="postgresql://{{username}}:{{password}}@postgres.my-app.svc:5432/mydb"   allowed_roles="my-app-role"   username="vault"   password="vault-admin-password"

# Create a role that generates credentials
vault write database/roles/my-app-role   db_name=my-postgres   creation_statements="CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO "{{name}}";"   default_ttl="1h"   max_ttl="24h"

Now annotate the pod to inject dynamic database credentials:

annotations:
  vault.hashicorp.com/agent-inject-secret-db-creds: "database/creds/my-app-role"
  vault.hashicorp.com/agent-inject-template-db-creds: |
    {{- with secret "database/creds/my-app-role" -}}
    DATABASE_URL=postgresql://{{ .Data.username }}:{{ .Data.password }}@postgres:5432/mydb
    {{- end }}

Each pod gets unique credentials. When the pod dies, the lease is revoked and those credentials stop working. Your database audit log shows exactly which pod made which queries.

Secrets Store CSI Driver as an alternative

If you prefer secrets as Kubernetes Secrets (for compatibility with existing tooling), the Secrets Store CSI Driver with the Vault provider is an alternative:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: my-app-vault
  namespace: my-app
spec:
  provider: vault
  parameters:
    vaultAddress: "http://vault.vault.svc:8200"
    roleName: "my-app"
    objects: |
      - objectName: "api-key"
        secretPath: "secret/data/my-app/config"
        secretKey: "api_key"
  secretObjects:
    - secretName: my-app-secrets
      type: Opaque
      data:
        - objectName: api-key
          key: API_KEY

This creates a Kubernetes Secret synced from Vault, which your pods can consume as normal env variables. Simpler than the agent injector for teams that want standard Kubernetes patterns.

Audit logging

Enable Vault's audit log — this is the main reason to choose Vault over AWS Secrets Manager for compliance-heavy environments:

vault audit enable file file_path=/vault/audit/audit.log

Every secret access is logged: timestamp, requesting entity, path accessed, and whether it was allowed or denied. For SOC 2 or ISO 27001, this is a significant compliance asset.

What not to do

• Don't put Vault's root token in a Kubernetes Secret. It defeats the purpose.
• Don't skip auto-unseal in production. Manual unseal after every restart is a 3 AM problem.
• Don't store Vault init keys in Git, even encrypted. Use a separate secure channel.
• Don't skip lease renewal — applications need to handle token and secret renewal or they'll stop working when leases expire.

Vault done right is a significant security upgrade. Vault done wrong (over-complex, poorly documented, manually operated) becomes an outage vector. Start simple, document everything, and automate the operational tasks from the beginning.