In modern containerized applications, managing sensitive information securely is crucial for maintaining application security. Kubernetes Secrets provide a native way to handle confidential data such as passwords, API tokens, and SSH keys. This comprehensive guide explores how to effectively manage secrets in Kubernetes environments.

Kubernetes Secrets are API objects specifically designed to hold sensitive information, keeping it separate from your application code and container configurations. This separation is fundamental to maintaining security best practices and compliance requirements in containerized environments.

This article will cover everything from basic concepts to advanced implementations, helping you understand and implement secure secret management in your Kubernetes clusters.

Understanding Kubernetes Secrets Basics

Definition and Purpose

Kubernetes Secrets are objects that store and manage sensitive information in your cluster. Unlike ConfigMaps, which handle general configuration data, Secrets are specifically designed for confidential information that requires additional protection.

Default Storage Mechanism

By default, Kubernetes stores secrets in its etcd database. While this provides a centralized location for secret management, it’s important to note that secrets are only base64 encoded by default, not encrypted.

Base64 Encoding vs Encryption

A common misconception is that base64 encoding provides security. It doesn’t – it’s merely an encoding method. For true security, you need to enable encryption at rest and implement proper access controls. We’ll cover encrypting secrets at rest below.

Core Use Cases

Kubernetes Secrets commonly store:

• Authentication credentials for applications
• API tokens for external service access
• TLS certificates for secure communication
• Database credentials
• OAuth tokens

Types of Kubernetes Secrets

Built-in Secret Types

Kubernetes provides several built-in secret types for common use cases:

Opaque Secrets

• The default and most flexible type
• Used for arbitrary user-defined data

apiVersion: v1
kind: Secret
metadata:
  name: opaque-secret-example
type: Opaque
data:
  username: YWRtaW4=  # "admin" base64 encoded
  password: cDQkJHcwcmQ=  # "p4$$w0rd" base64 encoded

Service Account Tokens

• Type: kubernetes.io/service-account-token
• Automatically created for service accounts

apiVersion: v1
kind: Secret
metadata:
  name: service-account-token-example
  annotations:
    kubernetes.io/service-account.name: "my-service-account"
type: kubernetes.io/service-account-token

Docker Registry Credentials

• Type: kubernetes.io/dockercfg (older format)
• Type: kubernetes.io/dockerconfigjson (newer format)
• Used for container registry authentication

# Newer format (recommended)
apiVersion: v1
kind: Secret
metadata:
  name: docker-registry-example
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5leGFtcGxlLmNvbSI6eyJ1c2VybmFtZSI6Im15dXNlciIsInBhc3N3b3JkIjoibXlwYXNzd29yZCIsImVtYWlsIjoibXllbWFpbEBleGFtcGxlLmNvbSIsImF1dGgiOiJiWGwxYzJWeU9tMTVjR0Z6YzNkdmNtUT0ifX19

# Command-line creation:
# kubectl create secret docker-registry docker-registry-example \
#   --docker-server=registry.example.com \
#   --docker-username=myuser \
#   --docker-password=mypassword \
#   --docker-email=myemail@example.com

TLS Certificates

• Type: kubernetes.io/tls
• Stores certificate and private key pairs

apiVersion: v1
kind: Secret
metadata:
  name: tls-secret-example
type: kubernetes.io/tls
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCi4uLiAoY2VydGlmaWNhdGUgZGF0YSkgLi4uCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  tls.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCi4uLiAocHJpdmF0ZSBrZXkgZGF0YSkgLi4uCi0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K

# Command-line creation:
# kubectl create secret tls tls-secret-example --cert=path/to/cert.pem --key=path/to/key.pem

Basic Authentication

• Type: kubernetes.io/basic-auth
• Stores credentials for basic authentication

apiVersion: v1
kind: Secret
metadata:
  name: basic-auth-example
type: kubernetes.io/basic-auth
data:
  username: YWRtaW4=  # "admin" base64 encoded
  password: cDQkJHcwcmQ=  # "p4$$w0rd" base64 encoded

SSH Authentication

• Type: kubernetes.io/ssh-auth
• Stores SSH credentials

apiVersion: v1
kind: Secret
metadata:
  name: ssh-auth-example
type: kubernetes.io/ssh-auth
data:
  ssh-privatekey: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQouLi4gKHByaXZhdGUga2V5IGRhdGEpIC4uLgotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

# Command-line creation:
# kubectl create secret generic ssh-auth-example \
#   --type=kubernetes.io/ssh-auth \
#   --from-file=ssh-privatekey=path/to/private-key

Custom Secret Types

You can create custom secret types by following the format:

<domain-name>/<secret-type>

For example: example.com/my-custom-secret

Here’s a complete example of defining and using a custom secret type:

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
type: example.com/database-credentials
data:
  host: bXlkYi5leGFtcGxlLmNvbQ==  # "mydb.example.com" base64 encoded
  username: YWRtaW4=              # "admin" base64 encoded
  password: cDQkJHcwcmQ=          # "p4$$w0rd" base64 encoded
  dbname: cHJvZHVjdGlvbg==        # "production" base64 encoded
  connectionTimeout: MzMA         # "30" base64 encoded

Custom secret types help with organizational clarity and can enable specialized controllers or operators to handle specific types of secrets in a customized manner. While Kubernetes itself treats custom secrets the same as Opaque secrets, your own controllers can implement type-specific validation or processing.

Working with Secrets in Kubernetes

Creating Secrets

Command-line Methods

# Create from literal values
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=secretpass

# Create from files
kubectl create secret generic tls-certs \
  --from-file=cert.pem \
  --from-file=key.pem

YAML Declaration

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  API_KEY: BASE64_ENCODED_VALUE
  API_SECRET: BASE64_ENCODED_VALUE

Accessing Secrets

Command Line

kubectl get secret [SECRET_NAME] -o jsonpath='{.data}'

Mounting as Files

volumes:
  - name: secrets-store
    secret:
      secretName: app-secrets
volumeMounts:
  - name: secrets-store
    mountPath: "/etc/secrets"
    readOnly: true

Using Environment Variables

env:
  - name: API_KEY
    valueFrom:
      secretKeyRef:
        name: app-secrets
        key: API_KEY

Enabling Encryption at Rest for Kubernetes Secrets

By default, Kubernetes only base64 encodes secrets in etcd, which isn’t true encryption. Here’s how to properly enable encryption at rest:

Create an encryption configuration file (encryption-config.yaml):

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: <base64-encoded-32-byte-key>
    - identity: {}

Generate a 32-byte random key and base64 encode it:

head -c 32 /dev/urandom | base64

Replace <base64-encoded-32-byte-key> in your encryption config with the generated key.

Configure the API server to use the encryption config by adding the following flag to the kube-apiserver manifest (typically found at /etc/kubernetes/manifests/kube-apiserver.yaml):

--encryption-provider-config=/etc/kubernetes/encryption-config.yaml

Mount the encryption config file into the API server pod:

volumes:
- name: encryption-config
  hostPath:
    path: /etc/kubernetes/encryption-config.yaml
    type: File
volumeMounts:
- name: encryption-config
  mountPath: /etc/kubernetes/encryption-config.yaml
  readOnly: true

Save the configuration. The API server will automatically restart.

To encrypt existing secrets, update them all by running:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

Verify encryption is working:

kubectl get secret <secret-name> -n <namespace> -o yaml

The secret data in etcd won’t be readable directly after encryption is properly configured.

Advanced Encryption Options

The major cloud providers also provide key management that can be leveraged for managing the encryption keys.

Using AWS KMS:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - kms:
        name: aws-kms
        endpoint: aws-kms://arn:aws:kms:<region>:<account>:key/<key-id>
        cachesize: 1000
        timeout: 3s
    - identity: {}

Using GCP KMS:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - kms:
        name: gcp-kms
        endpoint: gcp-kms://projects/<project>/locations/<location>/keyRings/<keyring>/cryptoKeys/<key>
        cachesize: 1000
        timeout: 3s
    - identity: {}

Using Azure Key Vault:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - kms:
        name: azure-kms
        endpoint: azurekms://<keyvault-name>.vault.azure.net/<key-name>
        cachesize: 1000
        timeout: 3s
    - identity: {}

Security Considerations and Best Practices

Default Security Measures

Kubernetes implements several security measures by default:

• Secrets are only distributed to nodes running pods that need them
• Secrets are stored in tmpfs on nodes
• Each secret is only sent to nodes that require it

Implementation Best Practices

Encryption at Rest

• Enable encryption for etcd
• Use KMS providers when possible
• Implement envelope encryption

RBAC Implementation

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: secret-reader
rules: rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]

Secret Rotation Strategies

• Implement automated rotation using tools like External Secrets Operator
• Set up monitoring for secret expiration
• Use short-lived credentials when possible

Namespace Isolation

• Use separate namespaces for different environments
• Implement network policies to restrict secret access
• Configure resource quotas for secrets

Common Challenges and Solutions

Known Limitations

1. Storage Constraints
   • 1MB maximum size per secret
   • Limited etcd performance with large secrets
2. Default Encryption Concerns
   • Base64 encoding isn’t encryption
   • Need to configure encryption at rest
3. Rotation Challenges
   • No built-in rotation mechanism
   • Manual rotation can be error-prone

Conclusion

Effective secret management in Kubernetes requires a comprehensive approach combining native features with additional tools and best practices. Key takeaways include:

• Always enable encryption at rest
• Implement proper RBAC controls
• Use external secret management solutions for advanced features
• Regular audit and rotation of secrets
• Maintain separate environments with appropriate controls