linux_patch_manager/docs/runbooks/key-management.md

Key Management Runbook

Applies to: Linux Patch Manager production deployments (issue #6 — secret encryption at rest) Last updated: 2026-06-03 Owner: SRE / Security

---

Overview

Linux Patch Manager uses two per-install AES-256-GCM encryption keys for protecting sensitive data at rest. Both keys are auto-generated on first start of the service, stored as 32-byte files with 0600 permissions (owner read/write only).

Key file	Path	Protects	Used by
health-check.key	/etc/patch-manager/keys/health-check.key	HTTP basic-auth passwords for health check endpoints	pm-web, pm-worker
secret-encryption.key	/etc/patch-manager/keys/secret-encryption.key	OIDC client_secret, SMTP smtp_password, TOTP totp_secret	pm-web, pm-auth, pm-worker

The two keys are separate by design (blast-radius isolation): if the health-check key is ever compromised, the app secrets remain protected by a different key.

---

Key Generation (First Start)

On first start of pm-web or pm-worker, the crypto::load_or_create_key() function checks for each key file. If missing, it:

1. Creates the /etc/patch-manager/keys/ directory (mode 0700)
2. Generates 32 cryptographically random bytes via OsRng (the OS CSPRNG)
3. Writes the key to disk
4. Sets permissions to 0600 (owner read/write only)
5. Returns the key to the calling code

The key files are created in the order they are first accessed. If pm-worker starts before pm-web, it creates the same key file (filesystem-shared). Both processes can read the same key.

---

Backup

Both key files MUST be included in /etc/patch-manager backups. Without the key files, encrypted data is unrecoverable. Recommended backup procedure:

# Include the keys directory in the backup archive
tar -czf /backup/patch-manager-$(date +%F).tar.gz \
    /etc/patch-manager/config.toml \
    /etc/patch-manager/keys/ \
    /var/lib/patch-manager/  # if used

# Verify the keys are in the backup
tar -tzf /backup/patch-manager-*.tar.gz | grep -E 'keys/.*\.key$'

The existing scripts/backup.sh already excludes secrets from unencrypted backups and supports GPG encryption for the archive. Ensure the backup includes the keys directory.

---

Verification (Production)

To verify both keys exist and have correct permissions on a running deployment:

# Check both key files exist with 0600 permissions
for key in health-check.key secret-encryption.key; do
    path="/etc/patch-manager/keys/${key}"
    if [ -f "$path" ]; then
        mode=$(stat -c '%a' "$path")
        size=$(stat -c '%s' "$path")
        echo "[OK]   $path  mode=$mode  size=$size"
    else
        echo "[FAIL] $path  missing"
    fi
done

Expected output:

[OK]   /etc/patch-manager/keys/health-check.key  mode=600  size=32
[OK]   /etc/patch-manager/keys/secret-encryption.key  mode=600  size=32

---

Recovery (Disaster Scenario)

If a key file is lost (disk failure, accidental deletion):

1. All encrypted data becomes unrecoverable. This includes:

   • HTTP basic-auth passwords for health check endpoints (health-check.key)
   • OIDC client_secret (secret-encryption.key)
   • SMTP smtp_password (secret-encryption.key)
   • TOTP totp_secret for all users (secret-encryption.key)

2. If you have a backup of the key files: restore them to /etc/patch-manager/keys/ with 0600 permissions. The service will read the restored keys on next start.

3. If you do NOT have a backup: re-provision the affected secrets:

   • For OIDC: re-enter the client_secret from the IdP's app registration
   • For SMTP: re-enter the SMTP password
   • For TOTP: all users must re-enroll MFA (their existing TOTP secrets are unrecoverable)
   • For health-check basic auth: re-enter the password in each health check configuration

---

Key Rotation

Key rotation is not yet supported (tracked as a follow-up issue). If a key is compromised:

1. Generate a new key: rm /etc/patch-manager/keys/secret-encryption.key (service will auto-generate on next start)
2. Re-encrypt all secrets in the database using the migrate-secrets binary (see README of the helper)
3. Update any external systems that depended on the old secrets (e.g., IdP app registration)

For a planned rotation (without compromise), the procedure is the same but coordinated with a maintenance window.

---

Security Notes

• Never log the key bytes or include them in error messages. The crypto::load_or_create_key() function returns the key but callers should never tracing::error! the value.
• Never commit key files to git. The /etc/patch-manager/keys/ directory should be in .gitignore or outside the repo entirely (recommended).
• Never copy key files between machines (e.g., for "easy migration"). Each deployment must generate its own key.
• The MASKED placeholder in API responses (e.g., for client_secret in OIDC settings) continues to apply on top of DB encryption — it's a separate defense-in-depth layer.

---

Related

• Secret encryption spec — full design rationale and migration plan
• Security review §4.1 — control matrix entry
• Migration 020 — schema changes for the new encrypted columns
• crates/pm-core/src/crypto.rs — implementation of load_or_create_key, encrypt, decrypt
• crates/migrate-secrets/src/main.rs — one-shot helper for migrating plaintext → encrypted