# Production Secrets & Configuration (/docs/deploying-agent-factories-in-the-cloud/90-real-cloud-deployment/06-production-secrets-configuration)

---
type: Document
title: "Production Secrets & Configuration"
description: "Manage secrets and configuration in cloud environments using Kubernetes Secrets, ConfigMaps, and cloud-native patterns"
status: stable
order: 9006
chapter: 90
lesson: 6
duration_minutes: 25
generated:
  by: ag-archive/0.1.0
  at: 2026-08-26T00:00:00Z
ksor:
  audience: [public]
  approval:
    by: ag-archive/0.1.0
    at: 2026-08-26T00:00:00Z
trust_tier: unverified
build_id: sha256:c8912c3008ae345a7fdc33ace656531ad74f37c065177057c390c7d759f1e970
source_commit: dad4758bb59919c7b9130444778ade062a2dce3e
ksor_version: 0.0.47
---

In Lesson 5, you deployed your Task API to DOKS with a basic configuration. But production services need more than hardcoded values. Your AI agent needs database credentials, API keys for external services, and environment-specific settings that change between staging and production.

Storing these directly in Deployment manifests creates security risks and operational headaches. Every configuration change requires a new container image or manifest edit. Worse, secrets end up in version control where anyone with repository access can read them.

Kubernetes solves this with two primitives: **Secrets** for sensitive data and **ConfigMaps** for non-sensitive configuration. This lesson teaches you to use both effectively, including the critical image pull secret that authenticates your cluster to pull images from GitHub Container Registry.

---

## What You'll Learn

This lesson covers 5 concepts organized into three groups:

| Concept Group | Concepts | Focus |
|---------------|----------|-------|
| **Secret Fundamentals** | 1-2 | Creating secrets, types of secrets (Opaque, docker-registry) |
| **Configuration Management** | 3-4 | ConfigMaps, configuration hierarchy patterns |
| **Verification** | 5 | Mounting secrets, verifying injection, troubleshooting |

**Prerequisites:**
- Lesson 5 completed (Task API deployed to DOKS)
- kubectl configured and connected to your cluster
- GitHub personal access token with `read:packages` scope

**Time Estimate:** 25 minutes
- Concepts: 15 minutes
- Exercises: 8 minutes
- Try With AI: 2 minutes

---

## Concept 1: Kubernetes Secrets

A Kubernetes Secret stores sensitive data like passwords, tokens, and certificates. Unlike ConfigMaps, Secrets are base64-encoded (not encrypted by default, but protected from casual viewing) and can be restricted with RBAC.

### Creating a Secret from Literal Values

```bash
kubectl create secret generic task-api-secrets \
  --from-literal=DATABASE_PASSWORD=prod-super-secret-password \
  --from-literal=OPENAI_API_KEY=sk-proj-your-api-key-here
```

**Output:**

```
secret/task-api-secrets created
```

### Viewing Secret Structure (Not Values)

```bash
kubectl get secret task-api-secrets -o yaml
```

**Output:**

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: task-api-secrets
  namespace: default
type: Opaque
data:
  DATABASE_PASSWORD: cHJvZC1zdXBlci1zZWNyZXQtcGFzc3dvcmQ=
  OPENAI_API_KEY: c2stcHJvai15b3VyLWFwaS1rZXktaGVyZQ==
```

The values are base64-encoded, not encrypted. Anyone with `kubectl get secret` access can decode them:

```bash
echo "cHJvZC1zdXBlci1zZWNyZXQtcGFzc3dvcmQ=" | base64 -d
```

**Output:**

```
prod-super-secret-password
```

**Security Note:** Base64 is encoding, not encryption. For true at-rest encryption, enable Kubernetes secrets encryption or use a secrets manager like HashiCorp Vault or cloud provider secrets (AWS Secrets Manager, GCP Secret Manager).

---

## Concept 2: Image Pull Secrets for Private Registries

Your Task API container lives in GitHub Container Registry (GHCR), which requires authentication. Without an image pull secret, Kubernetes cannot pull your private images.

### Creating the Image Pull Secret

```bash
kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=$GITHUB_USERNAME \
  --docker-password=$GITHUB_TOKEN \
  --docker-email=your-email@example.com
```

**Output:**

```
secret/ghcr-secret created
```

Where:
- `$GITHUB_USERNAME` is your GitHub username
- `$GITHUB_TOKEN` is a personal access token with `read:packages` scope

### Verifying the Secret

```bash
kubectl get secret ghcr-secret -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq .
```

**Output:**

```json
{
  "auths": {
    "ghcr.io": {
      "username": "your-username",
      "password": "ghp_xxxxxxxxxxxxxxxxxxxx",
      "email": "your-email@example.com",
      "auth": "eW91ci11c2VybmFtZTpnaHBfeHh4eHh4eHh4eHh4eHh4eHh4eHh4"
    }
  }
}
```

### Referencing in Deployment

Add `imagePullSecrets` to your Deployment spec:

```yaml
# task-api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: task-api
spec:
  template:
    spec:
      imagePullSecrets:
        - name: ghcr-secret
      containers:
        - name: task-api
          image: ghcr.io/your-username/task-api:latest
```

Without this reference, pod creation fails with `ImagePullBackOff`:

```bash
kubectl describe pod task-api-xxxxx | grep -A5 "Events:"
```

**Output (without secret):**

```
Events:
  Warning  Failed   12s  kubelet  Failed to pull image "ghcr.io/your-username/task-api:latest":
           rpc error: code = Unknown desc = failed to pull and unpack image:
           failed to resolve reference: unexpected status code 401 Unauthorized
```

---

## Concept 3: ConfigMaps for Non-Sensitive Configuration

ConfigMaps store configuration data that isn't sensitive: feature flags, service URLs, logging levels. Unlike Secrets, ConfigMaps appear in plain text and are easier to inspect.

### Creating a ConfigMap

```bash
kubectl create configmap task-api-config \
  --from-literal=LOG_LEVEL=info \
  --from-literal=API_BASE_URL=https://api.example.com \
  --from-literal=ENABLE_METRICS=true
```

**Output:**

```
secret/task-api-config created
```

### Creating ConfigMap from a File

For complex configuration, use a file:

```bash
# config.env
LOG_LEVEL=info
API_BASE_URL=https://api.example.com
ENABLE_METRICS=true
MAX_CONNECTIONS=100
REQUEST_TIMEOUT_MS=30000
```

```bash
kubectl create configmap task-api-config --from-env-file=config.env
```

**Output:**

```
configmap/task-api-config created
```

### Viewing ConfigMap Contents

```bash
kubectl get configmap task-api-config -o yaml
```

**Output:**

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: task-api-config
  namespace: default
data:
  API_BASE_URL: https://api.example.com
  ENABLE_METRICS: "true"
  LOG_LEVEL: info
  MAX_CONNECTIONS: "100"
  REQUEST_TIMEOUT_MS: "30000"
```

---

## Concept 4: Configuration Hierarchy Pattern

Production applications need layered configuration: defaults that work everywhere, environment overrides for staging vs production, and secrets that are never committed to version control.

### The Three-Layer Pattern

```
┌─────────────────────────────────────┐
│           Application               │
├─────────────────────────────────────┤
│  Layer 3: Secrets (highest priority)│  ← DATABASE_PASSWORD, API_KEY
│  kubectl create secret              │
├─────────────────────────────────────┤
│  Layer 2: ConfigMap (environment)   │  ← LOG_LEVEL, API_URL
│  kubectl create configmap           │
├─────────────────────────────────────┤
│  Layer 1: Container defaults        │  ← Baked into image
│  Dockerfile ENV statements          │
└─────────────────────────────────────┘
```

Environment variables from higher layers override lower layers.

### Implementing in Deployment

```yaml
# task-api-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: task-api
spec:
  template:
    spec:
      imagePullSecrets:
        - name: ghcr-secret
      containers:
        - name: task-api
          image: ghcr.io/your-username/task-api:latest

          # Layer 1: Defaults (optional, usually in Dockerfile)
          env:
            - name: NODE_ENV
              value: "production"

          # Layer 2: ConfigMap (all non-sensitive config)
          envFrom:
            - configMapRef:
                name: task-api-config

          # Layer 3: Secrets (sensitive values, highest priority)
            - secretRef:
                name: task-api-secrets
```

**Output (kubectl describe pod):**

```
Environment Variables from:
  task-api-config   ConfigMap  Optional: false
  task-api-secrets  Secret     Optional: false
Environment:
  NODE_ENV:  production
```

### Why This Order Matters

If `LOG_LEVEL` appears in both ConfigMap and as a hardcoded env var, the ConfigMap value wins because `envFrom` is processed after `env`. Secrets in `envFrom` are processed last, so they override everything.

---

## Concept 5: Verifying Secret and ConfigMap Injection

Deploying configuration is only half the battle. You must verify that containers receive the expected values.

### Check Pod Environment Variables

```bash
kubectl exec -it deployment/task-api -- env | grep -E "(DATABASE|OPENAI|LOG_LEVEL)"
```

**Output:**

```
DATABASE_PASSWORD=prod-super-secret-password
OPENAI_API_KEY=sk-proj-your-api-key-here
LOG_LEVEL=info
```

### Verify All Environment Variables

```bash
kubectl exec -it deployment/task-api -- printenv | sort
```

**Output:**

```
API_BASE_URL=https://api.example.com
DATABASE_PASSWORD=prod-super-secret-password
ENABLE_METRICS=true
LOG_LEVEL=info
MAX_CONNECTIONS=100
NODE_ENV=production
OPENAI_API_KEY=sk-proj-your-api-key-here
REQUEST_TIMEOUT_MS=30000
...
```

### Troubleshooting Missing Variables

If expected variables are missing, check:

1. **ConfigMap/Secret exists:**
   ```bash
   kubectl get configmap task-api-config
   kubectl get secret task-api-secrets
   ```

2. **Deployment references are correct:**
   ```bash
   kubectl get deployment task-api -o yaml | grep -A10 envFrom
   ```

3. **Pod events for errors:**
   ```bash
   kubectl describe pod -l app=task-api | grep -A20 Events
   ```

**Output (missing ConfigMap):**

```
Events:
  Warning  Failed  12s  kubelet  Error: configmap "task-api-config" not found
```

---

## Common Mistakes

**1. Forgetting imagePullSecrets for Private Registries**

```yaml
# WRONG - Missing imagePullSecrets
spec:
  containers:
    - name: task-api
      image: ghcr.io/your-username/task-api:latest  # Will fail with 401
```

**Fix:** Always include `imagePullSecrets` when using private registries:
```yaml
spec:
  imagePullSecrets:
    - name: ghcr-secret
  containers:
    - name: task-api
      image: ghcr.io/your-username/task-api:latest
```

**2. Putting Secrets in ConfigMaps**

```bash
# WRONG - API keys in ConfigMap (visible to anyone)
kubectl create configmap task-api-config \
  --from-literal=OPENAI_API_KEY=sk-proj-xxxxx
```

**Fix:** Use Secrets for sensitive data:
```bash
kubectl create secret generic task-api-secrets \
  --from-literal=OPENAI_API_KEY=sk-proj-xxxxx
```

**3. Using Wrong Secret Type for Docker Registry**

```bash
# WRONG - Opaque secret won't work for imagePullSecrets
kubectl create secret generic ghcr-secret \
  --from-literal=password=ghp_xxxxx
```

**Fix:** Use `docker-registry` type:
```bash
kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=$GITHUB_USERNAME \
  --docker-password=$GITHUB_TOKEN
```

**4. Forgetting to Restart Pods After Secret Update**

Updating a Secret doesn't automatically restart pods:

```bash
kubectl create secret generic task-api-secrets \
  --from-literal=DATABASE_PASSWORD=new-password \
  --dry-run=client -o yaml | kubectl apply -f -
# Pod still has OLD password!
```

**Fix:** Restart the deployment:
```bash
kubectl rollout restart deployment/task-api
```

---

## Exercise 6.1: Create Application Secrets

Create secrets for your Task API with database credentials and an API key.

**Instructions:**

1. Create the secret:
   ```bash
   kubectl create secret generic task-api-secrets \
     --from-literal=DATABASE_PASSWORD=my-secure-password \
     --from-literal=API_KEY=your-test-api-key
   ```

2. Verify creation:
   ```bash
   kubectl get secret task-api-secrets
   ```

**Expected output:** Secret listed with `Opaque` type and 2 data items.

---

## Exercise 6.2: Create Image Pull Secret for GHCR

Set up authentication for GitHub Container Registry.

**Instructions:**

1. Create a GitHub personal access token with `read:packages` scope at https://github.com/settings/tokens

2. Create the secret (replace with your values):
   ```bash
   export GITHUB_USERNAME=your-username
   export GITHUB_TOKEN=ghp_your_token_here

   kubectl create secret docker-registry ghcr-secret \
     --docker-server=ghcr.io \
     --docker-username=$GITHUB_USERNAME \
     --docker-password=$GITHUB_TOKEN
   ```

3. Verify:
   ```bash
   kubectl get secret ghcr-secret -o jsonpath='{.type}'
   ```

**Expected output:** `kubernetes.io/dockerconfigjson`

---

## Exercise 6.3: Create ConfigMap and Deploy with Full Configuration

Create a ConfigMap and update your deployment to use the three-layer configuration pattern.

**Instructions:**

1. Create ConfigMap:
   ```bash
   kubectl create configmap task-api-config \
     --from-literal=LOG_LEVEL=info \
     --from-literal=ENABLE_METRICS=true
   ```

2. Apply updated deployment with all references (save as `task-api-full.yaml`):
   ```yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: task-api
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: task-api
     template:
       metadata:
         labels:
           app: task-api
       spec:
         imagePullSecrets:
           - name: ghcr-secret
         containers:
           - name: task-api
             image: ghcr.io/your-username/task-api:latest
             ports:
               - containerPort: 8000
             envFrom:
               - configMapRef:
                   name: task-api-config
               - secretRef:
                   name: task-api-secrets
   ```

3. Apply and verify:
   ```bash
   kubectl apply -f task-api-full.yaml
   kubectl exec -it deployment/task-api -- printenv | grep -E "(LOG_LEVEL|DATABASE)"
   ```

**Expected output:** Both ConfigMap and Secret values visible in pod environment.

---

## Try With AI

**Part 1: Initial Request**

Your Task API needs additional secrets for a new feature: Redis connection string and a webhook signing secret. Ask AI for help:

"I need to add Redis and webhook secrets to my Kubernetes deployment. Create kubectl commands to add REDIS_URL and WEBHOOK_SECRET to my existing task-api-secrets."

**What you're learning:** How to extend existing secrets without recreating them from scratch.

**Part 2: Critical Evaluation**

Review AI's response. Ask yourself:
- Did it use `kubectl create secret` with `--dry-run=client -o yaml | kubectl apply -f -` to update existing secrets?
- Or did it suggest deleting and recreating (which causes downtime)?
- Does the approach preserve existing secret values?

**Part 3: Configuration Hierarchy**

Ask AI about a more advanced pattern:

"I want to use external-secrets operator to pull secrets from AWS Secrets Manager instead of creating them with kubectl. Is this better than what I'm doing now?"

**What you're learning:** The difference between simple kubectl-based secrets (good for learning) and enterprise patterns (external-secrets operator).

**Part 4: Validation**

Verify the secrets work by checking your pod:
```bash
kubectl exec -it deployment/task-api -- env | grep REDIS
```

If the variable appears with the expected value, your secret management is working.

---

## Reflect on Your Skill

You built a `multi-cloud-deployer` skill in Lesson 0. Test and improve it based on what you learned.

### Test Your Skill

```
Using my multi-cloud-deployer skill, set up secrets for a production deployment.
Does my skill include creating application secrets and image pull secrets?
```

### Identify Gaps

Ask yourself:
- Does my skill cover creating Kubernetes secrets with kubectl?
- Does it include image pull secrets for private registries like GHCR?
- Does it demonstrate the three-layer configuration pattern?
- Does it include verification steps using kubectl exec?

### Improve Your Skill

If you found gaps:

```
My multi-cloud-deployer skill is missing secrets management patterns.
Update it to include:
- Creating application secrets (kubectl create secret generic)
- Creating image pull secrets (kubectl create secret docker-registry)
- ConfigMap creation for non-sensitive configuration
- The three-layer hierarchy (defaults -> configmap -> secrets)
- Verification with kubectl exec to check environment variables
- Common mistakes like forgetting imagePullSecrets reference
```