fix(security): encrypt app secrets at rest with AES-256-GCM (#6)

Encrypt three sensitive secrets that were stored in plaintext: OIDC client_secret, SMTP smtp_password, TOTP totp_secret. AES-256-GCM via pm-core::crypto helper. New per-install key at /etc/patch-manager/keys/secret-encryption.key, separate from health-check.key for blast-radius isolation. MASKED placeholder behavior in API responses is preserved.

23 files changed, +1248 / -28. Closes #6.

docs/REST_API.md

@@ -119,6 +119,8 @@ Security: JWT Bearer Token (except Public Endpoints)
 | POST | `/settings/azure-sso/test` | Test Azure SSO compatibility |
 | POST | `/settings/audit-integrity` | Verify audit log integrity |
 
+> **Note (issue #6):** As of May 2026, sensitive fields (`oidc.client_secret`, `smtp.password`) are encrypted at rest in the database (AES-256-GCM). The `MASKED` placeholder behavior in API responses is **preserved** — clients never see plaintext secrets in GET responses. See [docs/runbooks/key-management.md](runbooks/key-management.md) for key management procedures.
+
 ## 12. Single Sign-On (SSO)
 | Method | Endpoint | Description |
 |--------|----------|-------------|

docs/runbooks/key-management.md

@@ -0,0 +1,128 @@
+# Key Management Runbook
+
+**Applies to:** Linux Patch Manager production deployments (issue #6 — secret encryption at rest)
+**Last updated:** 2026-06-03
+**Owner:** SRE / Security
+
+---
+
+## Overview
+
+Linux Patch Manager uses two per-install AES-256-GCM encryption keys for protecting sensitive data at rest. Both keys are auto-generated on first start of the service, stored as 32-byte files with `0600` permissions (owner read/write only).
+
+| Key file | Path | Protects | Used by |
+|----------|------|----------|---------|
+| `health-check.key` | `/etc/patch-manager/keys/health-check.key` | HTTP basic-auth passwords for health check endpoints | `pm-web`, `pm-worker` |
+| `secret-encryption.key` | `/etc/patch-manager/keys/secret-encryption.key` | OIDC `client_secret`, SMTP `smtp_password`, TOTP `totp_secret` | `pm-web`, `pm-auth`, `pm-worker` |
+
+The two keys are separate by design (blast-radius isolation): if the health-check key is ever compromised, the app secrets remain protected by a different key.
+
+---
+
+## Key Generation (First Start)
+
+On first start of `pm-web` or `pm-worker`, the `crypto::load_or_create_key()` function checks for each key file. If missing, it:
+
+1. Creates the `/etc/patch-manager/keys/` directory (mode `0700`)
+2. Generates 32 cryptographically random bytes via `OsRng` (the OS CSPRNG)
+3. Writes the key to disk
+4. Sets permissions to `0600` (owner read/write only)
+5. Returns the key to the calling code
+
+The key files are created in the order they are first accessed. If `pm-worker` starts before `pm-web`, it creates the same key file (filesystem-shared). Both processes can read the same key.
+
+---
+
+## Backup
+
+**Both key files MUST be included in `/etc/patch-manager` backups.** Without the key files, encrypted data is unrecoverable. Recommended backup procedure:
+
+```bash
+# Include the keys directory in the backup archive
+tar -czf /backup/patch-manager-$(date +%F).tar.gz \
+    /etc/patch-manager/config.toml \
+    /etc/patch-manager/keys/ \
+    /var/lib/patch-manager/  # if used
+
+# Verify the keys are in the backup
+tar -tzf /backup/patch-manager-*.tar.gz | grep -E 'keys/.*\.key$'
+```
+
+The existing `scripts/backup.sh` already excludes secrets from unencrypted backups and supports GPG encryption for the archive. Ensure the backup includes the keys directory.
+
+---
+
+## Verification (Production)
+
+To verify both keys exist and have correct permissions on a running deployment:
+
+```bash
+# Check both key files exist with 0600 permissions
+for key in health-check.key secret-encryption.key; do
+    path="/etc/patch-manager/keys/${key}"
+    if [ -f "$path" ]; then
+        mode=$(stat -c '%a' "$path")
+        size=$(stat -c '%s' "$path")
+        echo "[OK]   $path  mode=$mode  size=$size"
+    else
+        echo "[FAIL] $path  missing"
+    fi
+done
+```
+
+Expected output:
+```
+[OK]   /etc/patch-manager/keys/health-check.key  mode=600  size=32
+[OK]   /etc/patch-manager/keys/secret-encryption.key  mode=600  size=32
+```
+
+---
+
+## Recovery (Disaster Scenario)
+
+If a key file is lost (disk failure, accidental deletion):
+
+1. **All encrypted data becomes unrecoverable.** This includes:
+   - HTTP basic-auth passwords for health check endpoints (health-check.key)
+   - OIDC `client_secret` (secret-encryption.key)
+   - SMTP `smtp_password` (secret-encryption.key)
+   - TOTP `totp_secret` for all users (secret-encryption.key)
+
+2. **If you have a backup** of the key files: restore them to `/etc/patch-manager/keys/` with `0600` permissions. The service will read the restored keys on next start.
+
+3. **If you do NOT have a backup**: re-provision the affected secrets:
+   - For OIDC: re-enter the `client_secret` from the IdP's app registration
+   - For SMTP: re-enter the SMTP password
+   - For TOTP: all users must re-enroll MFA (their existing TOTP secrets are unrecoverable)
+   - For health-check basic auth: re-enter the password in each health check configuration
+
+---
+
+## Key Rotation
+
+Key rotation is **not yet supported** (tracked as a follow-up issue). If a key is compromised:
+
+1. Generate a new key: `rm /etc/patch-manager/keys/secret-encryption.key` (service will auto-generate on next start)
+2. Re-encrypt all secrets in the database using the `migrate-secrets` binary (see [README of the helper](../../crates/migrate-secrets/src/main.rs))
+3. Update any external systems that depended on the old secrets (e.g., IdP app registration)
+
+For a planned rotation (without compromise), the procedure is the same but coordinated with a maintenance window.
+
+---
+
+## Security Notes
+
+- **Never** log the key bytes or include them in error messages. The `crypto::load_or_create_key()` function returns the key but callers should never `tracing::error!` the value.
+- **Never** commit key files to git. The `/etc/patch-manager/keys/` directory should be in `.gitignore` or outside the repo entirely (recommended).
+- **Never** copy key files between machines (e.g., for "easy migration"). Each deployment must generate its own key.
+- **The `MASKED` placeholder in API responses** (e.g., for `client_secret` in OIDC settings) continues to apply on top of DB encryption — it's a separate defense-in-depth layer.
+
+---
+
+## Related
+
+- [Secret encryption spec](../../tasks/secret-encryption-spec.md) — full design rationale and migration plan
+- [Security review](../security-review.md) §4.1 — control matrix entry
+- [Migration 020](../../migrations/020_encrypt_secrets_at_rest.sql) — schema changes for the new encrypted columns
+- `crates/pm-core/src/crypto.rs` — implementation of `load_or_create_key`, `encrypt`, `decrypt`
+- `crates/migrate-secrets/src/main.rs` — one-shot helper for migrating plaintext → encrypted

docs/security-review.md

@@ -125,7 +125,8 @@ verifying that all mandated security controls are implemented and operational.
 | Control | Status | Evidence |
 |---------|--------|----------|
 | Infrastructure-managed disk encryption | ✅ Verified | Hardware/infrastructure layer provides encryption at rest; no LUKS in guest OS |
-| No column-level encryption needed | ✅ Verified | Compliance requirement satisfied by infrastructure layer per system mandate |
+| **App secrets encrypted at rest (issue #6 fix)** | ✅ Verified | OIDC `client_secret`, SMTP `smtp_password`, and TOTP `totp_secret` are encrypted with AES-256-GCM using a dedicated per-install key at `/etc/patch-manager/keys/secret-encryption.key` (auto-generated on first start, 0600 permissions). Separate from the health-check key for blast-radius isolation. Encryption/decryption via `pm-core::crypto::encrypt`/`decrypt`. Schema migration `020_encrypt_secrets_at_rest.sql` replaces plaintext TEXT columns with BYTEA `_encrypted` + `_nonce` columns. All 6 read/write sites updated: `sso.rs`, `settings.rs` (OIDC + SMTP), `session.rs` (TOTP read), `auth.rs` (TOTP write), `users.rs` (TOTP NULL), `pm-worker/email.rs` (SMTP read). The `MASKED` placeholder behavior in API responses is preserved. |
+| No column-level encryption needed | ❌ Superseded | Issue #6 (May 2026) introduced column-level encryption for app secrets. Updated to add app-secrets row above; other sensitive data continues to rely on the infrastructure layer. |
 
 ### 4.2 Secret Management
 | Control | Status | Evidence |