Production Secrets & Configuration

In Lesson 5, you deployed your Task API to DOKS with a basic configuration. But production services need more than hardcoded values. Your AI agent needs database credentials, API keys for external services, and environment-specific settings that change between staging and production.

Storing these directly in Deployment manifests creates security risks and operational headaches. Every configuration change requires a new container image or manifest edit. Worse, secrets end up in version control where anyone with repository access can read them.

Kubernetes solves this with two primitives: Secrets for sensitive data and ConfigMaps for non-sensitive configuration. This lesson teaches you to use both effectively, including the critical image pull secret that authenticates your cluster to pull images from GitHub Container Registry.

What You'll Learn

This lesson covers 5 concepts organized into three groups:

Prerequisites:

• Lesson 5 completed (Task API deployed to DOKS)
• kubectl configured and connected to your cluster
• GitHub personal access token with read:packages scope

Time Estimate: 25 minutes

• Concepts: 15 minutes
• Exercises: 8 minutes
• Try With AI: 2 minutes

Concept 1: Kubernetes Secrets

A Kubernetes Secret stores sensitive data like passwords, tokens, and certificates. Unlike ConfigMaps, Secrets are base64-encoded (not encrypted by default, but protected from casual viewing) and can be restricted with RBAC.

Creating a Secret from Literal Values

kubectl create secret generic task-api-secrets \
  --from-literal=DATABASE_PASSWORD=prod-super-secret-password \
  --from-literal=OPENAI_API_KEY=sk-proj-your-api-key-here

Output:

secret/task-api-secrets created

Viewing Secret Structure (Not Values)

kubectl get secret task-api-secrets -o yaml

Output:

apiVersion: v1
kind: Secret
metadata:
  name: task-api-secrets
  namespace: default
type: Opaque
data:
  DATABASE_PASSWORD: cHJvZC1zdXBlci1zZWNyZXQtcGFzc3dvcmQ=
  OPENAI_API_KEY: c2stcHJvai15b3VyLWFwaS1rZXktaGVyZQ==

The values are base64-encoded, not encrypted. Anyone with kubectl get secret access can decode them:

echo "cHJvZC1zdXBlci1zZWNyZXQtcGFzc3dvcmQ=" | base64 -d

Output:

prod-super-secret-password

Security Note: Base64 is encoding, not encryption. For true at-rest encryption, enable Kubernetes secrets encryption or use a secrets manager like HashiCorp Vault or cloud provider secrets (AWS Secrets Manager, GCP Secret Manager).

Concept 2: Image Pull Secrets for Private Registries

Your Task API container lives in GitHub Container Registry (GHCR), which requires authentication. Without an image pull secret, Kubernetes cannot pull your private images.

Creating the Image Pull Secret

kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=$GITHUB_USERNAME \
  --docker-password=$GITHUB_TOKEN \
  --docker-email=your-email@example.com

Output:

secret/ghcr-secret created

Where:

• $GITHUB_USERNAME is your GitHub username
• $GITHUB_TOKEN is a personal access token with read:packages scope

Verifying the Secret

kubectl get secret ghcr-secret -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .

Output:

{
  "auths": {
    "ghcr.io": {
      "username": "your-username",
      "password": "ghp_xxxxxxxxxxxxxxxxxxxx",
      "email": "your-email@example.com",
      "auth": "eW91ci11c2VybmFtZTpnaHBfeHh4eHh4eHh4eHh4eHh4eHh4eHh4"
    }
  }
}

Referencing in Deployment

Add imagePullSecrets to your Deployment spec:

# task-api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: task-api
spec:
  template:
    spec:
      imagePullSecrets:
        - name: ghcr-secret
      containers:
        - name: task-api
          image: ghcr.io/your-username/task-api:latest

Without this reference, pod creation fails with ImagePullBackOff:

kubectl describe pod task-api-xxxxx | grep -A5 "Events:"

Output (without secret):

Events:
  Warning  Failed   12s  kubelet  Failed to pull image "ghcr.io/your-username/task-api:latest":
           rpc error: code = Unknown desc = failed to pull and unpack image:
           failed to resolve reference: unexpected status code 401 Unauthorized

Concept 3: ConfigMaps for Non-Sensitive Configuration

ConfigMaps store configuration data that isn't sensitive: feature flags, service URLs, logging levels. Unlike Secrets, ConfigMaps appear in plain text and are easier to inspect.

Creating a ConfigMap

kubectl create configmap task-api-config \
  --from-literal=LOG_LEVEL=info \
  --from-literal=API_BASE_URL=https://api.example.com \
  --from-literal=ENABLE_METRICS=true

Output:

secret/task-api-config created

Creating ConfigMap from a File

For complex configuration, use a file:

# config.env
LOG_LEVEL=info
API_BASE_URL=https://api.example.com
ENABLE_METRICS=true
MAX_CONNECTIONS=100
REQUEST_TIMEOUT_MS=30000

kubectl create configmap task-api-config --from-env-file=config.env

Output:

configmap/task-api-config created

Viewing ConfigMap Contents

kubectl get configmap task-api-config -o yaml

Output:

apiVersion: v1
kind: ConfigMap
metadata:
  name: task-api-config
  namespace: default
data:
  API_BASE_URL: https://api.example.com
  ENABLE_METRICS: "true"
  LOG_LEVEL: info
  MAX_CONNECTIONS: "100"
  REQUEST_TIMEOUT_MS: "30000"

Concept 4: Configuration Hierarchy Pattern

Production applications need layered configuration: defaults that work everywhere, environment overrides for staging vs production, and secrets that are never committed to version control.

The Three-Layer Pattern

┌─────────────────────────────────────┐
│           Application               │
├─────────────────────────────────────┤
│  Layer 3: Secrets (highest priority)│  ← DATABASE_PASSWORD, API_KEY
│  kubectl create secret              │
├─────────────────────────────────────┤
│  Layer 2: ConfigMap (environment)   │  ← LOG_LEVEL, API_URL
│  kubectl create configmap           │
├─────────────────────────────────────┤
│  Layer 1: Container defaults        │  ← Baked into image
│  Dockerfile ENV statements          │
└─────────────────────────────────────┘

Environment variables from higher layers override lower layers.

Implementing in Deployment

# task-api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: task-api
spec:
  template:
    spec:
      imagePullSecrets:
        - name: ghcr-secret
      containers:
        - name: task-api
          image: ghcr.io/your-username/task-api:latest

          # Layer 1: Defaults (optional, usually in Dockerfile)
          env:
            - name: NODE_ENV
              value: "production"

          # Layer 2: ConfigMap (all non-sensitive config)
          envFrom:
            - configMapRef:
                name: task-api-config

          # Layer 3: Secrets (sensitive values, highest priority)
            - secretRef:
                name: task-api-secrets

Output (kubectl describe pod):

Environment Variables from:
  task-api-config   ConfigMap  Optional: false
  task-api-secrets  Secret     Optional: false
Environment:
  NODE_ENV:  production

Why This Order Matters

If LOG_LEVEL appears in both ConfigMap and as a hardcoded env var, the ConfigMap value wins because envFrom is processed after env. Secrets in envFrom are processed last, so they override everything.

Concept 5: Verifying Secret and ConfigMap Injection

Deploying configuration is only half the battle. You must verify that containers receive the expected values.

Check Pod Environment Variables

kubectl exec -it deployment/task-api -- env | grep -E "(DATABASE|OPENAI|LOG_LEVEL)"

Output:

DATABASE_PASSWORD=prod-super-secret-password
OPENAI_API_KEY=sk-proj-your-api-key-here
LOG_LEVEL=info

Verify All Environment Variables

kubectl exec -it deployment/task-api -- printenv | sort

Output:

API_BASE_URL=https://api.example.com
DATABASE_PASSWORD=prod-super-secret-password
ENABLE_METRICS=true
LOG_LEVEL=info
MAX_CONNECTIONS=100
NODE_ENV=production
OPENAI_API_KEY=sk-proj-your-api-key-here
REQUEST_TIMEOUT_MS=30000
...

Troubleshooting Missing Variables

If expected variables are missing, check:

1. ConfigMap/Secret exists:

   kubectl get configmap task-api-config
   kubectl get secret task-api-secrets

2. Deployment references are correct:

   kubectl get deployment task-api -o yaml | grep -A10 envFrom

3. Pod events for errors:

   kubectl describe pod -l app=task-api | grep -A20 Events

Output (missing ConfigMap):

Events:
  Warning  Failed  12s  kubelet  Error: configmap "task-api-config" not found

Common Mistakes

1. Forgetting imagePullSecrets for Private Registries

# WRONG - Missing imagePullSecrets
spec:
  containers:
    - name: task-api
      image: ghcr.io/your-username/task-api:latest  # Will fail with 401

Fix: Always include imagePullSecrets when using private registries:

spec:
  imagePullSecrets:
    - name: ghcr-secret
  containers:
    - name: task-api
      image: ghcr.io/your-username/task-api:latest

2. Putting Secrets in ConfigMaps

# WRONG - API keys in ConfigMap (visible to anyone)
kubectl create configmap task-api-config \
  --from-literal=OPENAI_API_KEY=sk-proj-xxxxx

Fix: Use Secrets for sensitive data:

kubectl create secret generic task-api-secrets \
  --from-literal=OPENAI_API_KEY=sk-proj-xxxxx

3. Using Wrong Secret Type for Docker Registry

# WRONG - Opaque secret won't work for imagePullSecrets
kubectl create secret generic ghcr-secret \
  --from-literal=password=ghp_xxxxx

Fix: Use docker-registry type:

kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=$GITHUB_USERNAME \
  --docker-password=$GITHUB_TOKEN

4. Forgetting to Restart Pods After Secret Update

Updating a Secret doesn't automatically restart pods:

kubectl create secret generic task-api-secrets \
  --from-literal=DATABASE_PASSWORD=new-password \
  --dry-run=client -o yaml | kubectl apply -f -
# Pod still has OLD password!

Fix: Restart the deployment:

kubectl rollout restart deployment/task-api

Exercise 6.1: Create Application Secrets

Create secrets for your Task API with database credentials and an API key.

Instructions:

1. Create the secret:

   kubectl create secret generic task-api-secrets \
     --from-literal=DATABASE_PASSWORD=my-secure-password \
     --from-literal=API_KEY=your-test-api-key

2. Verify creation:

   kubectl get secret task-api-secrets

Expected output: Secret listed with Opaque type and 2 data items.

Exercise 6.2: Create Image Pull Secret for GHCR

Set up authentication for GitHub Container Registry.

Instructions:

1. Create a GitHub personal access token with read:packages scope at https://github.com/settings/tokens

2. Create the secret (replace with your values):

   export GITHUB_USERNAME=your-username
   export GITHUB_TOKEN=ghp_your_token_here
   
   kubectl create secret docker-registry ghcr-secret \
     --docker-server=ghcr.io \
     --docker-username=$GITHUB_USERNAME \
     --docker-password=$GITHUB_TOKEN

3. Verify:

   kubectl get secret ghcr-secret -o jsonpath='{.type}'

Expected output: kubernetes.io/dockerconfigjson

Exercise 6.3: Create ConfigMap and Deploy with Full Configuration

Create a ConfigMap and update your deployment to use the three-layer configuration pattern.

Instructions:

1. Create ConfigMap:

   kubectl create configmap task-api-config \
     --from-literal=LOG_LEVEL=info \
     --from-literal=ENABLE_METRICS=true

2. Apply updated deployment with all references (save as task-api-full.yaml):

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: task-api
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: task-api
     template:
       metadata:
         labels:
           app: task-api
       spec:
         imagePullSecrets:
           - name: ghcr-secret
         containers:
           - name: task-api
             image: ghcr.io/your-username/task-api:latest
             ports:
               - containerPort: 8000
             envFrom:
               - configMapRef:
                   name: task-api-config
               - secretRef:
                   name: task-api-secrets

3. Apply and verify:

   kubectl apply -f task-api-full.yaml
   kubectl exec -it deployment/task-api -- printenv | grep -E "(LOG_LEVEL|DATABASE)"

Expected output: Both ConfigMap and Secret values visible in pod environment.

Try With AI

Part 1: Initial Request

Your Task API needs additional secrets for a new feature: Redis connection string and a webhook signing secret. Ask AI for help:

"I need to add Redis and webhook secrets to my Kubernetes deployment. Create kubectl commands to add REDIS_URL and WEBHOOK_SECRET to my existing task-api-secrets."

What you're learning: How to extend existing secrets without recreating them from scratch.

Part 2: Critical Evaluation

Review AI's response. Ask yourself:

• Did it use kubectl create secret with --dry-run=client -o yaml | kubectl apply -f - to update existing secrets?
• Or did it suggest deleting and recreating (which causes downtime)?
• Does the approach preserve existing secret values?

Part 3: Configuration Hierarchy

Ask AI about a more advanced pattern:

"I want to use external-secrets operator to pull secrets from AWS Secrets Manager instead of creating them with kubectl. Is this better than what I'm doing now?"

What you're learning: The difference between simple kubectl-based secrets (good for learning) and enterprise patterns (external-secrets operator).

Part 4: Validation

Verify the secrets work by checking your pod:

kubectl exec -it deployment/task-api -- env | grep REDIS

If the variable appears with the expected value, your secret management is working.

Reflect on Your Skill

You built a multi-cloud-deployer skill in Lesson 0. Test and improve it based on what you learned.

Test Your Skill

Using my multi-cloud-deployer skill, set up secrets for a production deployment.
Does my skill include creating application secrets and image pull secrets?

Identify Gaps

Ask yourself:

• Does my skill cover creating Kubernetes secrets with kubectl?
• Does it include image pull secrets for private registries like GHCR?
• Does it demonstrate the three-layer configuration pattern?
• Does it include verification steps using kubectl exec?

Improve Your Skill

If you found gaps:

My multi-cloud-deployer skill is missing secrets management patterns.
Update it to include:
- Creating application secrets (kubectl create secret generic)
- Creating image pull secrets (kubectl create secret docker-registry)
- ConfigMap creation for non-sensitive configuration
- The three-layer hierarchy (defaults -> configmap -> secrets)
- Verification with kubectl exec to check environment variables
- Common mistakes like forgetting imagePullSecrets reference

Core Concept

Configure production secrets securely in the cloud using external stores and scoped access, ensuring the Task API and supporting services only receive what they need.

Key Mental Models

• Externalize and scope: Secrets live in managers (not manifests) with least-privilege access.
• Separation by env: Different namespaces/projects get distinct secrets and credentials.
• Secure injection: Deliver secrets at deploy time via CSI/ESO with auditability.

Critical Patterns

• Set up External Secrets/CSI to pull from cloud secret stores; map to app namespaces.
• Lock down access with RBAC/service identities and component scopes.
• Rotate and audit secrets; avoid logging or shipping them with images.

Common Mistakes

• Sharing secrets across environments or storing them in Git/Helm values.
• Over-permissive secret access for service accounts.
• Failing to rotate credentials or test secret updates in running pods.

Connections

• Builds on: DO cluster/app deployment; security practices from Chapter 88.
• Leads to: Checklist verification and capstone production release.