Secrets Management Runbook¶

This repo uses SOPS + age for secrets at rest. Private keys are stored in 1Password; encrypted files live in git alongside the code they configure.

Architecture¶

┌─────────────────────────────────────────────────────┐
│  .sops.yaml (in repo)                               │
│  Lists age public keys as recipients                │
└──────────┬──────────────────────────┬───────────────┘
           │                          │
    ┌──────▼──────┐           ┌───────▼──────┐
    │  Operator    │           │  CI (GitHub  │
    │  age key     │           │  Actions)    │
    │  (1Password) │           │  age key     │
    └──────┬──────┘           └───────┬──────┘
           │                          │
           ▼                          ▼
    sops -d secrets/foo.sops.yaml    sops -d (in workflow)

Bootstrap (new admin machine)¶

# 1. Sign in to 1Password CLI
op signin

# 2. Fetch the age key from 1Password and install it
./scripts/bootstrap-secrets.sh

# 3. Verify
sops -d secrets/example/demo.sops.yaml

The key is written to ~/.config/sops/age/keys.txt (the standard sops location). SOPS finds it automatically — no env vars needed.

Windows + Cursor (`@signageos/vscode-sops`)¶

Typical setup on this repo: WSL holds the age key (via bootstrap-secrets.sh); Windows holds sops (winget). The VS Code extension runs on Windows and must point at both explicitly.

Item	This machine (example)
`sops.exe`	`%LOCALAPPDATA%\Microsoft\WinGet\Packages\Mozilla.SOPS_Microsoft.Winget.Source_8wekyb3d8bbwe\sops.exe`
Age key	`\\wsl.localhost\Ubuntu\home\someone\.config\sops\age\keys.txt`

Cursor user settings (or gitignored .vscode/settings.json):

{
  "sops.enabled": true,
  "sops.binPath": "C:\\Users\\<you>\\AppData\\Local\\Microsoft\\WinGet\\Packages\\Mozilla.SOPS_Microsoft.Winget.Source_8wekyb3d8bbwe\\sops.exe",
  "sops.beta": false,
  "sops.creationEnabled": false,
  "sops.defaults.ageKeyFile": "\\\\wsl.localhost\\Ubuntu\\home\\someone\\.config\\sops\\age\\keys.txt",
  "files.associations": {
    "*.sops.yaml": "yaml"
  }
}

After changing settings: Developer: Reload Window (no full Cursor restart required).

Still seeing ENC[...] ciphertext?

Confirm the setting is sops.enabled (not sops.enable).
Bottom-right language mode must be YAML — .env.sops.yaml is often mis-detected as dotenv; force "*.sops.yaml": "yaml" in files.associations.
Close the tab and reopen the file (extension hooks on editor activation).
Check the status bar for SOPS: toggle enc/dec file — click it if you land on the encrypted tab; the extension opens a sibling .decrypted~.env.sops.yaml view.
View → Output → SOPS (or Developer Tools console) for decrypt errors.
Run SOPS: SOPS info from the command palette to confirm the extension is active.

The extension writes .decrypted~* scratch files next to the encrypted file; those are gitignored.

PowerShell in Cursor can't find sops? Integrated terminals inherit PATH from when Cursor started. After winget install Mozilla.SOPS, either restart Cursor or refresh the session:

. .\scripts\sops-env.ps1
sops -d services/monitoring/.env.sops.yaml

Git decrypted diffs (optional):

Cursor/Git for Windows runs textconv in Git Bash, which does not see winget PATH. Use the repo wrapper (not bare sops):

git config --global diff.sopsdiffer.textconv "bash $(pwd -W 2>/dev/null || pwd)/scripts/git-sops-textconv.sh"

From PowerShell in the repo root:

git config --global diff.sopsdiffer.textconv "bash $(Resolve-Path scripts/git-sops-textconv.sh)"

If diffs still fail, unset: git config --global --unset diff.sopsdiffer.textconv (you will see ciphertext in diffs again).

Repo .gitattributes marks *.sops.yaml and secrets/** for sopsdiffer.

On managed hosts¶

Managed hosts store the automation age key at /etc/homelab/age-key.txt. Non-interactive sessions (SSH, scripts, systemd) must set the key file path explicitly:

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d <file>

For interactive shells, add this to the user's profile (.bashrc, .zshrc):

export SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt

The key file is owned by root:docker with mode 440, so users in the docker group can read it.

First-time setup (generating keys)¶

Only run this once, when creating the repo's encryption keys:

# Generate operator age key and store in 1Password
./scripts/setup-age-key.sh

# The script prints the public key. Add it to .sops.yaml:
#   OPERATOR_AGE_PUBLIC_KEY → replace with your actual public key

# Generate CI key (for GitHub Actions)
age-keygen -o /tmp/ci-age-key.txt
# Copy the public key to .sops.yaml as CI_AGE_PUBLIC_KEY
# Copy the private key to GitHub repo secret: SOPS_AGE_KEY
# Then delete /tmp/ci-age-key.txt

Encrypting a new secret¶

# Create and encrypt in one step (opens $EDITOR)
sops secrets/services/my-service.sops.yaml

# Or encrypt an existing plaintext file
sops -e -i my-plaintext.yaml

SOPS uses the creation rules in .sops.yaml to determine which keys to use based on the file path.

Tailscale manual keys: see secrets/tailscale/README.md (files under secrets/tailscale/*.sops.yaml).

Decrypting¶

# Decrypt to stdout
sops -d secrets/services/my-service.sops.yaml

# Decrypt in place (careful — don't commit the plaintext!)
sops -d -i secrets/services/my-service.sops.yaml

Editing an encrypted file¶

# Opens in $EDITOR, re-encrypts on save
sops secrets/services/my-service.sops.yaml

On infra-services (no operator age key on your laptop)¶

If you do not have the operator age private key on Windows yet, you can still edit SOPS files on a managed host that already has /etc/homelab/age-key.txt (e.g. SSH to infra-services, use a clone such as /opt/homelab where git push works for you):

ssh someone@192.168.6.17
cd /opt/homelab
export SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt
sops infra/ansible/inventory/group_vars/tailscale/tailscale.sops.yaml
# edit, save, exit — file stays encrypted on disk except briefly in the editor
git add infra/ansible/inventory/group_vars/tailscale/tailscale.sops.yaml
git commit -m "chore: rotate Tailscale auth key in SOPS"
git push

This is expected and safe as long as only ciphertext is committed. “Don’t decrypt on the VM” means don’t leave plaintext secrets in the repo or paste private material into logs — not that sops must never run there.

Key rotation¶

If a key is compromised or an operator leaves:

Generate a new age key: age-keygen
Update .sops.yaml with the new public key
Remove the compromised key from .sops.yaml
Re-encrypt all files with the new key set:

# Find all SOPS files and update their recipients
find secrets/ -name '*.sops.yaml' -exec sops updatekeys {} \;
find services/ -name '.env.sops.yaml' -exec sops updatekeys {} \;

Commit the re-encrypted files
Store the new private key in 1Password (replace the old one)
If the CI key was compromised, rotate the GitHub Actions secret too

Compromise response¶

If you suspect a key has been leaked:

Immediately rotate the affected key (see above)
Assume all secrets encrypted with that key are compromised
Rotate all downstream credentials (API tokens, passwords, etc.)
Check git history — if plaintext was ever committed, consider the repo's full history compromised. Use git filter-repo to scrub.
Update the security register: docs/security-register.md

Decrypting SOPS YAML to dotenv¶

SOPS YAML files use key: value format, but Docker Compose .env files expect KEY=VALUE. When decrypting a SOPS file to a .env for a Compose stack, convert the format:

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d .env.sops.yaml \
  | sed 's/: /=/' > .env

If the SOPS key name differs from the environment variable the service expects, chain an additional sed substitution. For example, Traefik expects CLOUDFLARE_DNS_API_TOKEN but the SOPS file uses the shorter CF_DNS_API_TOKEN:

SOPS_AGE_KEY_FILE=/etc/homelab/age-key.txt sops -d .env.sops.yaml \
  | sed 's/CF_DNS_API_TOKEN: /CLOUDFLARE_DNS_API_TOKEN=/' > .env

File conventions¶

Path pattern	Purpose
`secrets/ansible/*.sops.yaml`	Ansible vault replacements
`secrets/services/<id>/*.sops.yaml`	Per-service credentials
`secrets/appliances/<id>/*.sops.yaml`	Appliance configs (e.g., Saltbox)
`services/<id>/.env.sops.yaml`	Compose stack environment variables
`infra/ansible/inventory/group_vars/all/backup.sops.yaml`	Restic backup credentials (B2, restic password)
`infra/ansible/inventory/group_vars/tailscale/tailscale.sops.yaml`	Ansible `tailscale` role: `tailscale_auth_key` with only `tag:server` (see Phase 7 §5-pre-a).
`secrets/tailscale/nas.sops.yaml`	Manual Synology: `tailscale_auth_key` with only `tag:nas` (Phase 7 §5-pre-b).
`secrets/tailscale/customer-app.sops.yaml`	Manual VMs: `tailscale_auth_key` with only `tag:customer-app` (Phase 7 §5-pre-b).

Secrets inventory¶

File	Keys	Consumer
`infra/ansible/inventory/group_vars/tailscale/tailscale.sops.yaml`	`tailscale_auth_key`	Ansible `tailscale` role (`tag:server` only — Phase 7 §5-pre-a)
`secrets/tailscale/nas.sops.yaml`	`tailscale_auth_key`	Manual whrrr (`tag:nas` only — Phase 7 §5-pre-b)
`secrets/tailscale/customer-app.sops.yaml`	`tailscale_auth_key`	Manual recordurbate / ubuncap (`tag:customer-app` only — Phase 7 §5-pre-b)
`services/traefik/.env.sops.yaml`	`CF_DNS_API_TOKEN`	Traefik (Cloudflare DNS-01 cert resolver)
`services/monitoring/.env.sops.yaml`	`DISCORD_WEBHOOK_URL`, `GF_SECURITY_ADMIN_PASSWORD`	Alertmanager (Discord), Grafana admin login
`services/authentik-outpost/.env.sops.yaml`	`AUTHENTIK_TOKEN`	Authentik proxy outpost on infra-services (ADR-002)
`services/komodo/secrets.sops.yaml`	`KOMODO_DATABASE_PASSWORD`, `KOMODO_INIT_ADMIN_`, `KOMODO_WEBHOOK_SECRET`, `KOMODO_JWT_SECRET`, `KOMODO_OIDC_CLIENT_`	Komodo — rendered to gitignored `compose.env` via `scripts/render-komodo-compose-env.sh`
`infra/ansible/inventory/group_vars/all/backup.sops.yaml`	`backup_restic_password`, `backup_b2_*`	Ansible `backup-client` role (`sops -d` at apply time, same pattern as `tailscale` SOPS)

Saltbox integration¶

Saltbox expects two config files on disk:

File	Source (encrypted)	Destination on `saltierpoop`
`accounts.yml`	`secrets/appliances/saltierpoop/accounts.sops.yaml`	`/srv/git/saltbox/accounts.yml`
`settings.yml`	`secrets/appliances/saltierpoop/settings.sops.yaml`	`/srv/git/saltbox/settings.yml`

These are decrypted by Ansible during the saltbox-host role (Phase 3) and placed at the paths Saltbox expects. The role uses sops -d and writes the plaintext to the target host. The plaintext never exists in git.

CI integration¶

GitHub Actions uses a dedicated age key (separate from the operator key) so that CI can validate encrypted files without having access to the operator's 1Password.

GitHub Secret	Value
`SOPS_AGE_KEY`	The CI age private key (full content of the key file)

The CI workflow sets SOPS_AGE_KEY as an environment variable. SOPS reads it automatically (no key file needed).

Pre-commit hook¶

The sops-encryption-check hook in .pre-commit-config.yaml verifies that all *.sops.yaml files contain the sops metadata key — catching accidental commits of plaintext files that should have been encrypted.