Kubernetes Secrets Management in 2026: Best Practices

Someone pushes a small change late on a Friday. Maybe it's a config update, a quick hotfix, or a troubleshooting tweak that wasn't meant to stay. Buried in the commit is a database connection string or API key.

A few minutes later, someone notices the mistake and removes it.

The problem is that deleting the file doesn't undo the exposure.

Public repositories are constantly monitored by scanners looking for leaked credentials. By the time a bad commit is reverted, the secret may already be copied and stored elsewhere.

A Kubernetes Secret is a built-in object used to store sensitive values like passwords, API keys, tokens, and certificates. But Secrets are not automatically secure. By default, they are base64-encoded, not encrypted.

That is why secrets management is an operational problem, not just a deployment task. At scale, teams need clear controls for access, rotation, auditing, and incident response.

Why Secrets Management Gets Harder at Scale

Managing three application secrets in a single namespace is not difficult. Managing secrets across 200 services, 15 teams, and four environments is a completely different problem.

Secret Sprawl

Secret sprawl happens when secrets multiply without any central inventory or ownership. A team creates a Secret for a new service, another team copies it into a different namespace for convenience, a third team hardcodes the same value in a ConfigMap "just for testing," and suddenly the same database password exists in seven different places across the cluster.

When that password needs to be rotated, nobody knows where all the copies are. The rotation breaks two services nobody expected. This is secret sprawl. It is one of the most common operational problems in growing Kubernetes platforms.

No Single Source of Truth

Many teams end up with secrets stored in multiple systems at the same time - some in Kubernetes Secrets, some in HashiCorp Vault, some in AWS Secrets Manager, and some in environment variable files on developer laptops. When there is no single source of truth, it is impossible to answer basic audit questions like "when was this credential last changed?" or "which pods currently have access to this secret?"

Short-Lived vs Long-Lived Credentials

A long-lived credential stored in a Kubernetes Secret is a permanently open attack window. If the etcd database is ever compromised, every static password and API key stored in it is exposed. Production platforms are moving toward short-lived, automatically rotating credentials - but this requires operational infrastructure that does not come out of the box with Kubernetes.

How Kubernetes Secrets Actually Work

Most documentation explains what a Kubernetes Secret is. Very few explain the full lifecycle of a secret from creation to use inside a container. Understanding this lifecycle is essential for securing it.

The most important stage to understand is the etcd storage step. This is where most teams have a false sense of security.

Why Base64 Is Not Encryption

The single biggest misconception about Kubernetes Secrets is that they are encrypted. They are not  by default.

Kubernetes stores Secret values encoded in base64. Base64 is a text encoding format, not a security mechanism. Any engineer with access to the cluster can decode a Secret in seconds:

# This is all it takes to read a "protected" Kubernetes Secret
kubectl get secret my-db-credentials \
  -o jsonpath='{.data.password}' | base64 -d

This returns the plaintext password immediately. There is no key required, no decryption step, and no access log entry for the decode operation.

At the storage layer, it is even more direct. Without encryption at rest configured, etcd stores Secrets as plain base64 strings in its key-value database. If an attacker gains access to the etcd volume  through a misconfigured backup, a compromised node, or a snapshot left in a storage bucket  every Secret in the cluster is immediately readable.

Encrypting Secrets at Rest in etcd

Encrypting Secrets at rest means that even if someone gets direct access to the etcd database, the values are unreadable without the decryption key. This is a non-negotiable configuration for any cluster handling production credentials.

EncryptionConfiguration

Kubernetes supports encryption at rest through an EncryptionConfiguration resource that is passed to the API Server at startup. The simplest provider is AES-CBC with a local key, but the production-standard approach is using a KMS provider so the encryption key itself is stored outside the cluster.

Basic EncryptionConfiguration using AES-CBC:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {} # fallback for reading unencrypted secrets

KMS Providers - The Production Standard

Storing the encryption key inside the cluster defeats the purpose of encrypting Secrets. The production approach is using a KMS envelope encryption provider. The cluster holds only an encrypted version of the data encryption key (DEK). The actual key encryption key (KEK) lives in a managed KMS service outside the cluster.

RBAC and Secret Access Control

Encryption protects data at rest. RBAC controls who can read, write, and list Secrets in a running cluster. Both are required. A cluster can have perfect encryption but completely open RBAC - and an insider or compromised service account can still read everything.

The Least Privilege Rule

The default behavior in many clusters is to grant too much access. Service accounts get cluster-wide permissions because it is faster to configure. Developers get admin-level RBAC because it is easier than scoping permissions properly. Over time, the blast radius of any compromised credential grows.

The correct approach is to scope every role to the minimum access required:

# A Role that allows access to only one specific Secret
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-db-secret
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    # restricted to this one Secret only
    resourceNames: ["db-credentials"]
    # get only - no list, no watch, no update
    verbs: ["get"]

Auditing Who Can Access Secrets

Before tightening RBAC, it helps to understand the current state. These commands reveal the actual access levels that exist today:

# Check if the current context can read Secrets in production
kubectl auth can-i get secrets -n production

# Check if a specific service account can list all Secrets
kubectl auth can-i list secrets -n production \
  --as=system:serviceaccount:production:app-service-account

# Find Roles and ClusterRoles that grant access to Secrets
# Grep on Roles - not RoleBindings - because permissions live in the Role itself
kubectl get clusterroles,roles -A -o yaml | grep -A5 -B5 secrets

# For a complete picture, rakkess and kubectl-who-can are the production-standard tools
# rakkess: https://github.com/corneliusweig/rakkess
# kubectl-who-can: https://github.com/aquasecurity/kubectl-who-can

Namespace Boundaries

Kubernetes RBAC is namespace-scoped for Roles, but ClusterRoles apply across all namespaces. A common mistake is using ClusterRoleBindings for service accounts that only need access within a single namespace. This silently grants the service account access to Secrets across every namespace in the cluster.

Secret Rotation Strategies

A Secret that is never rotated is a permanently open door. If a credential is compromised and nobody knows - which is often the case - it stays compromised indefinitely. Rotation limits the blast radius of any exposure.

What Needs Rotating and How Often 

The Rotation Challenge

Manual credential rotation sounds simple but breaks down at scale. If a database password is referenced in 15 different Kubernetes Secrets across 4 namespaces, rotating it means updating all 15 objects, restarting all affected pods, and verifying nothing broke. A missed reference takes down an application.

The solution is centralizing the source of truth. If all 15 services read the credential from the same Vault path or AWS Secrets Manager entry, rotating it in one place propagates automatically. This is the primary operational argument for using an external secret manager rather than storing credentials directly in Kubernetes Secrets.

External Secret Managers

Native Kubernetes Secrets solve the storage problem but not the lifecycle problem. They do not handle rotation, versioning, audit trails, or cross-cluster synchronization. External secret managers fill this gap.

External Secrets Operator (ESO)

The External Secrets Operator is the most widely adopted bridge between Kubernetes and external secret stores. It runs as a controller inside the cluster and automatically creates and updates Kubernetes Secrets by pulling values from an external store on a configurable sync interval.

ESO ExternalSecret pointing to AWS Secrets Manager:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: prod/database/credentials
        property: password

When the password is rotated in AWS Secrets Manager, ESO picks up the new value within one hour and updates the Kubernetes Secret automatically. No manual kubectl edits required.

Comparing the Main Options

The practical decision for most teams: if the cluster is on a single cloud provider, use that provider's native secret manager with ESO as the bridge into Kubernetes. If the platform spans multiple clouds or needs dynamic database credentials, HashiCorp Vault is worth the operational investment.

ESO vs Secrets Store CSI Driver

ESO is not the only integration path. Secrets Store CSI Driver can mount secrets directly into pods from an external secret store without creating a Kubernetes Secret object, unless optional Secret sync is enabled. This can reduce the Kubernetes Secret object attack surface, but it does not remove all secret exposure risk because the secret is still available inside the running pod.

For most teams, ESO is simpler to operate and sufficient. The CSI Driver is worth considering for high-security environments where the presence of a Kubernetes Secret object - even briefly - is unacceptable, such as handling private keys or regulatory credentials.

Hidden Ways Secrets Leak

Encryption at rest and tight RBAC protect secrets from direct theft. But secrets regularly leak through indirect paths that are much harder to detect.

Application Logs

The most common source of secret leakage in production is application logs. A developer adds a debug log statement that prints the full database connection string. The app ships to production. Every log collection agent (Fluentd, Datadog, Loki) scrapes and stores that connection string permanently in the logging backend.

The fix is never logging values read from environment variables or mounted Secret files without explicit scrubbing. Structured logging frameworks can help by restricting which fields are serialized.

Environment Variables

Injecting Secrets as environment variables is the easiest Kubernetes pattern, but it has a significant operational risk. Environment variables are readable by every process inside the container. If the container runs a shell, the full environment - including all credentials - is visible with a single env command.

More critically, many crash reporting tools, APM agents, and debugging tools automatically capture and report the environment of a crashing process. A single unhandled exception can send all credentials to an external crash reporting service.

CI/CD Pipelines

Many teams inject Kubernetes Secrets into CI/CD pipelines to run integration tests or deploy to staging. Pipeline logs are often stored for 90 days and accessible to any engineer with pipeline access. A secret printed to a pipeline log by a failing build step is exposed to everyone who can view that pipeline.

Every CI/CD tool (GitHub Actions, GitLab CI, Jenkins, Argo Workflows) has a secret masking feature that prevents values from appearing in logs. These masking configurations must be treated as a deployment requirement, not an optional configuration.

Note that this problem is primarily a CI concern, not a CD concern. In pull-based GitOps platforms using ArgoCD or Flux, the CD controller runs inside the cluster and does not need credentials passed through pipeline logs. Secret synchronization is handled entirely by in-cluster controllers like ESO. The leakage risk through pipeline logs is eliminated when CD pipelines do not touch secrets at all.

Shell History

Engineers who run kubectl create secret generic with the --from-literal flag pass credentials directly in the terminal command. That command lands in the shell history file (.bash_history or .zsh_history). If the engineer's laptop is compromised, or if the history file is accidentally included in a Docker build context, the credential leaks.

# This lands in shell history - do NOT use in production
kubectl create secret generic db-pass --from-literal=password=mysecretpassword

The real fix is not a safer kubectl command - it is removing manual secret creation from production entirely. In platforms that use an external secret manager as the source of truth, no engineer should be running kubectl create secret in production. Kubernetes admission controllers such as OPA Gatekeeper or Kyverno can enforce this at the API level, rejecting any Secret creation that does not originate from the approved ESO controller service account.

GitOps Secret Workflows

Teams using GitOps pipelines with ArgoCD or Flux often encrypt secrets before committing them to Git using tools like SOPS or Sealed Secrets. This is the right pattern. But the encryption is only as strong as the key management around it.

• If the SOPS master key or age key is stored as a Kubernetes Secret without proper RBAC, any workload with Secret access can decrypt the entire GitOps secret store.
• Sealed Secrets encrypts for a specific cluster using the controller's private key. If that controller key is not backed up and the cluster is destroyed, all sealed secrets become permanently unreadable - and teams sometimes store plaintext copies just in case.
• ArgoCD and Flux require bootstrap credentials to access the Git repository. These credentials, which often have full repository read access, are stored in cluster Secrets and are high-value targets.

The leak pattern in GitOps environments is usually not the encrypted file in Git - it is the decryption key stored insecurely, or the bootstrap credentials left over-provisioned for months without rotation.

Kubernetes Audit Logs

Kubernetes audit logs capture every API Server request including secret access events. If your audit policy logs Secrets at the RequestResponse level, the audit log can include request or response bodies and may expose Secret values. For Secret resources, prefer Metadata level logging so access is recorded without storing the Secret payload. This means the audit log itself can contain plaintext Secret values. Audit policy must be explicitly configured to omit response bodies for Secret resources.

Multi-Tenant Kubernetes Clusters

In a multi-tenant cluster where multiple teams share the same Kubernetes control plane, secret isolation between tenants becomes a critical security boundary. A secret belonging to Team A must be completely inaccessible to Team B - even if both teams have cluster access.

Namespace Isolation

Kubernetes Secrets are namespace-scoped. A Secret in namespace team-a is not accessible from namespace team-b through the Kubernetes API. But namespace isolation alone is not sufficient - RBAC must be configured correctly, and ClusterAdmin-level access must not be granted broadly.

A service account in team-a should never have ClusterRoleBindings that allow it to read secrets across all namespaces. This is a surprisingly common misconfiguration in clusters that were set up quickly.

Audit Logging for Multi-Tenant Clusters

In a shared cluster, audit logs become essential for demonstrating tenant isolation to security teams. The following audit policy captures all Secret access events without storing the actual values:

# Audit policy: log secret access metadata but not values
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]

With this policy, every kubectl get secret call generates an audit record showing who accessed which Secret and when - without the audit log containing the actual secret value.

Network Policies and Secret Exposure

A pod that can make outbound network calls can exfiltrate secrets even if RBAC is configured correctly. A compromised application that has been granted Secret access can read credentials and send them to an external endpoint. Network Policies should restrict outbound traffic from sensitive workloads to only the endpoints they actually need.

Kubernetes Secrets Management Checklist

Use this checklist when reviewing a Kubernetes cluster's secrets posture. A production cluster should be able to answer "yes" to every item.

• Encryption at rest is enabled: EncryptionConfiguration is applied and existing Secrets have been re-encrypted.
• KMS envelope encryption is used: the encryption key lives outside the cluster in AWS KMS, Azure Key Vault, or Google Cloud KMS.
• RBAC is scoped to least privilege: no service account has cluster-wide Secret list/get unless it genuinely requires it.
• Service accounts use namespace-scoped Roles, not ClusterRoleBindings.
• An external secret manager is used as the source of truth: Vault, AWS Secrets Manager, or equivalent.
• Secret rotation is automated: no long-lived static credentials exist in production.
• TLS certificates are managed by cert-manager and rotate automatically.
• Short-lived projected ServiceAccountTokens are used instead of long-lived tokens.
• Secrets are mounted as volume files, not environment variables, for sensitive credentials.
• Audit logging is enabled and covers all Secret access events.
• CI/CD pipelines use secret masking: no credential values appear in pipeline logs.
• kubectl create secret --from-literal is prohibited in production workflows.
• No Secrets exist in Git repositories, including deleted commits.
• Secret access is regularly reviewed and unused permissions are removed.

Troubleshooting Kubernetes Secrets Problems

When something breaks with secrets in Kubernetes, the error messages are often misleading. A pod stuck in CreateContainerConfigError, a failing readiness probe, or an application reporting "permission denied" can all trace back to a secrets misconfiguration. Here is how to work through the most common scenarios.

Pod Fails to Start - Missing Secret Reference

If a pod fails with CreateContainerConfigError or Error: secret not found, the pod spec references a Secret that does not exist in that namespace.

# See the exact error - look for "secret not found" in Events
kubectl describe pod <pod-name> -n <namespace>

# List all Secrets in the namespace to check what actually exists
kubectl get secrets -n <namespace>

Check that the namespace in the Secret spec matches the namespace of the pod. Secrets are namespace-scoped. A Secret in the default namespace is not accessible to a pod in production.

Pod Running But App Using Old Credential

This is the rotation trap described earlier. ESO has updated the Kubernetes Secret but the application is still using the old value.

# Verify the Kubernetes Secret has the new value
kubectl get secret db-credentials -n production \
  -o jsonpath='{.data.password}' | base64 -d

# If the pod was not restarted since the Secret was updated,
# env var secrets still hold the old value - trigger a rollout
kubectl rollout restart deployment/<deployment-name> -n production

ESO Not Syncing

If the ExternalSecret object exists but the Kubernetes Secret is not being created or updated, check the ESO controller logs and the ExternalSecret status field.

# Check sync status and error messages on the ExternalSecret object
kubectl describe externalsecret db-credentials -n production

# Check ESO controller logs for auth failures or API errors
kubectl logs -n external-secrets \
  deployment/external-secrets -f | grep -i error

The most common ESO failures are: IAM permission errors (the ESO service account cannot read from AWS Secrets Manager), wrong secret path in the remoteRef, or an expired authentication token.

RBAC Denial on Secret Access

If an application logs "secrets is forbidden: User cannot get resource secrets", the service account does not have permission.

# Check exactly what permissions the service account has
kubectl auth can-i get secret db-credentials \
  -n production \
  --as=system:serviceaccount:production:app-service-account

If the command returns no, create the appropriate Role and RoleBinding. If it returns yes but the app still gets denied, check that the app is actually running as that service account with kubectl describe pod.

Common Mistakes Teams Can Avoid in 2026

Hardcoded Credentials in Container Images

Credentials baked into Docker images at build time are one of the most persistent security problems in Kubernetes. Even if the image is stored in a private registry, anyone who can pull the image can extract the credential with docker inspect or by exploring the image layers. Credentials must never appear in Dockerfiles or in build arguments that get cached in image layers.

Shared Service Accounts Across Applications

A single Kubernetes service account used by fifteen different applications is fifteen times the blast radius of a single compromised application. If any one of those applications is breached, the attacker inherits the permissions of the shared service account across all fifteen. Every application needs its own service account with scoped permissions.

No Rotation Policy

Many teams create a production database password once during initial setup and never rotate it. The credential exists in the Kubernetes Secret, potentially in Vault, possibly in a developer's notes, and almost certainly in an old Slack message. Without a documented and automated rotation schedule, the credential effectively never changes.

Cluster-Admin Grants to Developers

Granting cluster-admin to developers for "convenience" or "debugging" is one of the fastest ways to guarantee a future breach. Cluster-admin can read every Secret across every namespace in the cluster. In a 200-service production cluster, that means every database password, every API key, and every TLS private key is accessible to anyone holding a cluster-admin binding.

Ignoring Secret Access in Audit Logs

Kubernetes audit logging is often enabled but never reviewed. Audit logs that nobody reads are operationally useless. Teams should set up automated alerts for unusual secret access patterns - for example, a service account that normally reads two Secrets suddenly reading fifteen, or a Secret being accessed from an unusual source IP.

Frequently Asked Questions(FAQs)

Are Kubernetes Secrets encrypted by default?

No. Kubernetes Secrets are base64-encoded by default, not encrypted. Base64 only changes how the value is represented; it does not protect the secret. To protect Secrets at rest, encryption must be explicitly enabled through EncryptionConfiguration and preferably backed by a KMS provider.

Where are Kubernetes Secrets stored?

Kubernetes Secrets are stored in etcd, the key-value database used by the Kubernetes control plane. This is why etcd encryption, backup security, and access control are important. If etcd snapshots or backups are exposed without encryption, secret values can be recovered.

Should Kubernetes Secrets be stored in Git?

Plaintext Kubernetes Secrets should never be stored in Git. If a team uses GitOps, secrets should be encrypted before being committed using tools such as SOPS or Sealed Secrets. The encryption keys must be protected carefully, because leaking the key can expose the encrypted secret store.

What is the safest way to manage Kubernetes Secrets in production?

The safest production pattern is to use an external secret manager such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or Google Secret Manager as the source of truth. Kubernetes should receive only the secrets required by workloads, with strict RBAC, encryption at rest, rotation, and audit logging enabled.

What is the difference between External Secrets Operator and Secrets Store CSI Driver?

External Secrets Operator syncs values from an external secret store into Kubernetes Secret objects. This is easier to adopt because most applications already know how to read Kubernetes Secrets as environment variables or mounted files. Secrets Store CSI Driver takes a different approach: it can mount secrets directly into pods from the external store without creating a Kubernetes Secret object, unless optional Secret sync is enabled. This can reduce the Kubernetes Secret attack surface, but it also adds more moving parts because the CSI driver, provider plugin, node integration, and application reload behavior all need to be managed carefully.

Can a running pod automatically use an updated Secret?

Not always. If a Secret is used as an environment variable, the pod usually needs to be restarted. If the Secret is mounted as a file, Kubernetes can update the file, but the application still needs to reread or reload it. Secret rotation should always include a rollout or reload strategy.