Secrets Management in Kubernetes: Best Practices Guide
Kubernetes has become the default orchestration layer for enterprise workloads, but its native secrets handling remains one of the platform's most misunderstood capabilities. A Kubernetes Secret object stored without encryption at rest is merely base64-encoded — not encrypted. For enterprises operating regulated workloads, this distinction is the difference between compliance and exposure. This guide covers the full spectrum of Kubernetes secrets management, from hardening native primitives to deploying enterprise-grade external secret operators.
The Problem with Native Kubernetes Secrets
Kubernetes Secrets are first-class API objects designed to hold sensitive data such as passwords, tokens, and TLS certificates. However, their default behavior introduces several security concerns that enterprise teams must address.
Base64 Is Not Encryption
By default, Kubernetes stores Secret objects in etcd as base64-encoded strings. Any user or process with read access to etcd — or with sufficient RBAC permissions to query the Kubernetes API — can decode these values trivially. Base64 is an encoding scheme, not an encryption algorithm.
Secrets in etcd
Unless you explicitly enable encryption at rest, every secret in your cluster is stored in plaintext (base64) in etcd. A compromised etcd backup, a misconfigured snapshot policy, or an attacker with node-level access can extract every secret in the cluster in seconds.
RBAC Overpermission
Many cluster configurations grant overly broad RBAC permissions that allow namespaced or cluster-wide secret enumeration. A service account that can list secrets in a namespace can read every credential in that namespace, regardless of whether the associated pod needs those credentials.
Essential Hardening: Non-Negotiable Steps
Before evaluating external solutions, every enterprise Kubernetes deployment should implement these baseline controls.
1. Enable Encryption at Rest
Configure the Kubernetes API server with an EncryptionConfiguration that encrypts Secret objects before they are written to etcd. Use AES-CBC or, preferably, AES-GCM with a KMS provider for key management.
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- kms:
apiVersion: v2
name: aws-kms-provider
endpoint: unix:///var/run/kmsplugin/socket.sock
- identity: {}
With KMS integration, the encryption key itself is managed by your cloud provider's key management service (AWS KMS, Azure Key Vault, GCP Cloud KMS), ensuring that even a full etcd compromise does not expose secret values without access to the KMS key.
2. Restrict RBAC for Secrets
Apply the principle of least privilege to secret access. No service account should have list or watch permissions on secrets unless absolutely required. Prefer get permissions on specific named secrets.
- Avoid cluster-wide
ClusterRolebindings that grant secret access across all namespaces - Use namespace isolation to limit the blast radius of a compromised service account
- Audit RBAC policies regularly with tools like
kubectl-who-canor Kubernetes audit logs
3. Enable Audit Logging
Configure Kubernetes audit policies to log every secret access event at the RequestResponse level. Ship these logs to your SIEM for anomaly detection. A pod that suddenly begins reading secrets it has never accessed before is a strong compromise indicator.
4. Disable Auto-Mounting of Service Account Tokens
By default, Kubernetes mounts a service account token into every pod. If a pod does not need to interact with the Kubernetes API, set automountServiceAccountToken: false in the pod spec to eliminate an unnecessary credential from the workload.
External Secrets Management Approaches
For enterprises requiring stronger guarantees, external secrets management systems provide capabilities that native Kubernetes Secrets cannot.
| Approach | How It Works | Best For |
|---|---|---|
| External Secrets Operator | Syncs secrets from external stores (Vault, AWS SM, GCP SM) into K8s Secret objects | Teams wanting a Kubernetes-native workflow with external source of truth |
| Vault Sidecar Injector | Injects secrets directly into pod filesystems via a mutating webhook and sidecar container | Organizations already running HashiCorp Vault |
| Sealed Secrets | Encrypts secrets client-side for safe storage in Git; decrypted only by the cluster controller | GitOps workflows where secrets must live in version control |
| CSI Secrets Store Driver | Mounts secrets from external stores as volumes using the Container Storage Interface | Workloads requiring file-based secret access without K8s Secret objects |
| SPIFFE/SPIRE | Issues short-lived X.509 certificates as workload identities | Zero-trust service mesh deployments with mTLS requirements |
External Secrets Operator: Deep Dive
The External Secrets Operator (ESO) has emerged as the most widely adopted approach for enterprise Kubernetes deployments. ESO defines two custom resources: SecretStore (which configures the connection to your external secrets provider) and ExternalSecret (which declares which secrets to sync and how).
Key Benefits
- Single source of truth — Secrets are managed in your external vault; Kubernetes receives synchronized copies
- Automatic refresh — ESO polls the external store at configurable intervals and updates the K8s Secret object when the source changes
- Multi-provider support — Works with AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HashiCorp Vault, and more
- Kubernetes-native — Applications consume standard K8s Secrets; no application code changes required
Rotation Strategy
Configure ESO with a refreshInterval that matches your rotation policy. When a secret is rotated in the external store, ESO updates the corresponding K8s Secret within the refresh window. Combine this with pod restart policies or file-watching sidecars to ensure running applications pick up the new credential.
Vault Sidecar Injector Pattern
HashiCorp's Vault Agent Injector uses a Kubernetes mutating admission webhook to automatically inject a Vault Agent sidecar container into annotated pods. The sidecar authenticates with Vault using the pod's Kubernetes service account token, retrieves the requested secrets, and writes them to a shared in-memory volume.
Advantages Over K8s Secrets
- Secrets never exist as Kubernetes Secret objects — they bypass etcd entirely
- Dynamic secrets with automatic lease renewal
- Secrets are stored in tmpfs (memory-backed), not on disk
- Fine-grained Vault policies control which pods can access which secrets
Operational Considerations
The sidecar pattern adds a container to every pod that needs secrets, increasing resource consumption. The Vault Agent requires network connectivity to the Vault cluster, creating a dependency that must be accounted for in availability planning. If Vault is unreachable during pod startup, the pod will fail to start.
Sealed Secrets for GitOps
For organizations practicing GitOps — where the entire cluster state is defined in Git — storing secrets in version control is unavoidable. Sealed Secrets addresses this by providing a client-side encryption tool (kubeseal) that encrypts secrets with the cluster's public key. Only the Sealed Secrets controller running in the cluster possesses the private key to decrypt them.
Sealed Secrets solves the specific problem of storing encrypted secrets in Git repositories. It does not replace a full secrets management platform for rotation, auditing, or dynamic credential generation.
Zero-Trust Workload Identity with SPIFFE
The most forward-looking approach to Kubernetes secrets management eliminates static secrets entirely. SPIFFE (Secure Production Identity Framework for Everyone) provides cryptographic workload identities — short-lived X.509 certificates issued to each workload based on attestation of its runtime properties. Combined with mTLS, workloads authenticate to each other without passwords, tokens, or API keys.
SPIRE (the SPIFFE Runtime Environment) serves as the identity provider, attesting workloads based on Kubernetes namespace, service account, pod label, and node properties. Certificates are automatically rotated with TTLs as short as one hour.
Enterprise Best Practices Checklist
- Enable encryption at rest for all Kubernetes Secret objects using a KMS provider
- Restrict RBAC permissions for secrets to the minimum necessary scope
- Implement an external secrets management solution (ESO, Vault, or CSI driver)
- Never store unencrypted secrets in Git repositories, Helm values, or CI/CD configurations
- Enable Kubernetes audit logging for all secret access events
- Rotate secrets automatically on a defined schedule (30 days maximum for production credentials)
- Use namespace isolation to limit the blast radius of credential compromise
- Disable auto-mounting of service account tokens for pods that do not need API access
- Monitor for anomalous secret access patterns in your SIEM
- Test secret rotation procedures in staging before production deployment
Kubernetes Secrets, Enterprise-Grade
Keys.yachts integrates natively with Kubernetes via External Secrets Operator and CSI driver. One control plane for every cluster, every cloud.
Explore the PlatformConclusion
Native Kubernetes Secrets are a starting point, not a destination. Enterprise deployments require encryption at rest, strict RBAC, external secret stores, automated rotation, and comprehensive audit logging. The approach you choose — ESO, Vault sidecar, Sealed Secrets, or SPIFFE — depends on your existing infrastructure, compliance requirements, and operational maturity. What matters is that secrets management is treated as a first-class concern in your Kubernetes architecture, not an afterthought bolted on after the first audit finding.