Secrets Management Runbook¶

This repo uses SOPS + age for secrets at rest. Private keys are stored in 1Password; encrypted files live in git alongside the code they configure.

Architecture¶

┌─────────────────────────────────────────────────────┐
│  .sops.yaml (in repo)                               │
│  Lists age public keys as recipients                │
└──────────┬──────────────────────────┬───────────────┘
           │                          │
    ┌──────▼──────┐           ┌───────▼──────┐
    │  Operator    │           │  CI (GitHub  │
    │  age key     │           │  Actions)    │
    │  (1Password) │           │  age key     │
    └──────┬──────┘           └───────┬──────┘
           │                          │
           ▼                          ▼
    sops -d secrets/foo.sops.yaml    sops -d (in workflow)

Bootstrap (new admin machine)¶

# 1. Sign in to 1Password CLI
op signin

# 2. Fetch the age key from 1Password and install it
./scripts/bootstrap-secrets.sh

# 3. Verify
sops -d secrets/example/demo.sops.yaml

The key is written to ~/.config/sops/age/keys.txt (the standard sops location). SOPS finds it automatically — no env vars needed.

Windows + Cursor (`@signageos/vscode-sops`)¶

Typical setup on this repo: WSL holds the age key (via bootstrap-secrets.sh); Windows holds sops (winget). The VS Code extension runs on Windows and must point at both explicitly.

Item	This machine (example)
`sops.exe`	`%LOCALAPPDATA%\Microsoft\WinGet\Packages\Mozilla.SOPS_Microsoft.Winget.Source_8wekyb3d8bbwe\sops.exe`
Age key	`\\wsl.localhost\Ubuntu\home\someone\.config\sops\age\keys.txt`

Cursor user settings (or gitignored .vscode/settings.json):

{
  "sops.enabled": true,
  "sops.binPath": "C:\\Users\\<you>\\AppData\\Local\\Microsoft\\WinGet\\Packages\\Mozilla.SOPS_Microsoft.Winget.Source_8wekyb3d8bbwe\\sops.exe",
  "sops.beta": false,
  "sops.creationEnabled": false,
  "sops.defaults.ageKeyFile": "\\\\wsl.localhost\\Ubuntu\\home\\someone\\.config\\sops\\age\\keys.txt",
  "files.associations": {
    "*.sops.yaml": "yaml"
  }
}

After changing settings: Developer: Reload Window (no full Cursor restart required).

Still seeing ENC[...] ciphertext?

Confirm the setting is sops.enabled (not sops.enable).
Bottom-right language mode must be YAML — .env.sops.yaml is often mis-detected as dotenv; force "*.sops.yaml": "yaml" in files.associations.
Close the tab and reopen the file (extension hooks on editor activation).
Check the status bar for SOPS: toggle enc/dec file — click it if you land on the encrypted tab; the extension opens a sibling .decrypted~.env.sops.yaml view.
View → Output → SOPS (or Developer Tools console) for decrypt errors.
Run SOPS: SOPS info from the command palette to confirm the extension is active.

The extension writes .decrypted~* scratch files next to the encrypted file; those are gitignored.

PowerShell in Cursor can't find sops? Integrated terminals inherit PATH from when Cursor started. After winget install Mozilla.SOPS, either restart Cursor or refresh the session:

. .\scripts\sops-env.ps1
sops -d services/monitoring/.env.sops.yaml

Git decrypted diffs (optional):

Cursor/Git for Windows runs textconv in Git Bash, which does not see winget PATH. Use the repo wrapper (not bare sops):

git config --global diff.sopsdiffer.textconv "bash $(pwd -W 2>/dev/null || pwd)/scripts/git-sops-textconv.sh"

From PowerShell in the repo root:

git config --global diff.sopsdiffer.textconv "bash $(Resolve-Path scripts/git-sops-textconv.sh)"

If diffs still fail, unset: git config --global --unset diff.sopsdiffer.textconv (you will see ciphertext in diffs again).

Repo .gitattributes marks *.sops.yaml and secrets/** for sopsdiffer.

On managed hosts¶

Bootstrap installs the automation age key at /etc/homelab/age-key.txt as root:root 0400. Do not assume membership in docker grants access, and do not chmod/chown the key to make a command work. Actual target modes must be checked separately by the authorized owner; this is the source contract.

Use an explicitly authorized root operation for that key, for example the noninteractive sudo -n render recipes below. If sudo authority is unavailable, stop rather than prompt, change permissions or deploy stale values. An operator with a separately provisioned readable age key can instead run the helper without sudo and pass --age-key /absolute/operator/key; the output will be owned by that operator. Never pass private key material or decrypted values in command arguments.

First-time setup (generating keys)¶

Only run this once, when creating the repo's encryption keys:

# Generate operator age key and store in 1Password
./scripts/setup-age-key.sh

# The script prints the public key. Add it to .sops.yaml:
#   OPERATOR_AGE_PUBLIC_KEY → replace with your actual public key

# Generate CI key (for GitHub Actions)
age-keygen -o /tmp/ci-age-key.txt
# Copy the public key to .sops.yaml as CI_AGE_PUBLIC_KEY
# Copy the private key to GitHub repo secret: SOPS_AGE_KEY
# Then delete /tmp/ci-age-key.txt

Encrypting a new secret¶

# Create and encrypt in one step (opens $EDITOR)
sops secrets/services/my-service.sops.yaml

# Or encrypt an existing plaintext file
sops -e -i my-plaintext.yaml

SOPS uses the creation rules in .sops.yaml to determine which keys to use based on the file path.

Committed *.sops.yaml files must contain encrypted scalar values. The sops-encryption-check pre-commit/CI guard fails files that only have stale SOPS metadata but plaintext values outside the sops metadata block. null values are allowed for intentionally empty fields.

Tailscale manual keys: see secrets/tailscale/README.md (files under secrets/tailscale/*.sops.yaml).

Decrypting¶

# Decrypt to stdout
sops -d secrets/services/my-service.sops.yaml

# Decrypt in place (careful — don't commit the plaintext!)
sops -d -i secrets/services/my-service.sops.yaml

Editing an encrypted file¶

# Opens in $EDITOR, re-encrypts on save
sops secrets/services/my-service.sops.yaml

On infra-services (no operator age key on your laptop)¶

Editing on a managed host requires separate authorization for the root-only automation key and a trusted editor/checkout. It is not made possible merely by SSH login as someone. Prefer editing with a separately provisioned operator key on the operator machine. If neither authorized context is available, stop; do not copy the host key or weaken its permissions. Commit only the encrypted source after checking the staged paths; do not include decrypted views or runtime env files. Key editing/rotation is not authorized by the rendering recipe.

Key rotation¶

If a key is compromised or an operator leaves:

Generate a new age key: age-keygen
Update .sops.yaml with the new public key
Remove the compromised key from .sops.yaml
Re-encrypt all files with the new key set:

# Find all SOPS files and update their recipients
find secrets/ -name '*.sops.yaml' -exec sops updatekeys {} \;
find services/ -name '.env.sops.yaml' -exec sops updatekeys {} \;

Commit the re-encrypted files
Store the new private key in 1Password (replace the old one)
If the CI key was compromised, rotate the GitHub Actions secret too

Compromise response¶

If you suspect a key has been leaked:

Immediately rotate the affected key (see above)
Assume all secrets encrypted with that key are compromised
Rotate all downstream credentials (API tokens, passwords, etc.)
Check git history — if plaintext was ever committed, consider the repo's full history compromised. Use git filter-repo to scrub.
Update the security register: docs/security-register.md

Decrypting SOPS YAML to dotenv¶

Use a maintained service renderer where one exists (Firefly and Komodo have specialized contracts). For other flat SOPS YAML mappings, use scripts/render-sops-env.py from the reviewed full checkout. Do not redirect SOPS into a working env file, use a fixed .env.tmp, or pipe into sed. A failed render must stop the surrounding shell before deployment.

Prerequisites and authority¶

Python 3.10+ and SOPS on the executing identity's PATH; no package installation is performed by the helper. Firefly's manual --restricted mode checks this interpreter version and the shared helper before secret access. Existing automatic Firefly callers do not opt in and retain the shell/SOPS backend; its Python migration is explicitly deferred, not assumed available in Periphery. Komodo already required Python; it now also requires the shared helper file.
Existing destination parents, no symlink directories/destination or hardlinked destination, and a reviewed checkout whose writers are trusted. Running root code from a checkout writable by an untrusted identity is unsafe. The helper is not a sandbox against hostile directory writers or source changes during a render. Do not run concurrent source editing, renders or deployments.
The default age key is /etc/homelab/age-key.txt, unless --age-key or SOPS_AGE_KEY_FILE explicitly selects an authorized alternative. No key modes are changed. Root renders produce root-owned 0600 .env; run Compose in that same authorized context. Operator-key renders produce operator-owned 0600 output. Coordinate any different automation consumer before cutover, not by making plaintext group/world-readable.
Verify .env and .decrypted~* remain untracked and ignored in this checkout. Never use git add -f, stage runtime plaintext, run shell tracing, print config output containing real values, or capture decrypted diagnostics in evidence.

Example from /opt/homelab/services/authentik-outpost, after separate deployment approval (the command itself only renders):

set -euo pipefail
cd /opt/homelab/services/authentik-outpost || exit 1
sudo -n python3 /opt/homelab/scripts/render-sops-env.py \
  --source .env.sops.yaml --output .env --require AUTHENTIK_TOKEN || exit 1
# Only after render success and the separate deployment gate:
sudo -n docker compose up -d

The helper decrypts to JSON in memory and suppresses SOPS diagnostics, rejects failed/invalid conversion, and validates every value before promotion. List every nonempty required output key with repeated --require. YAML values must be strings (quote booleans/numbers); duplicate keys, nested values, missing/empty required values and REPLACE_ME / CHANGE_ME / SET_FROM_AUTHENTIK / GENERATE_ON_HOST placeholders fail. Optional empty strings are allowed; application-specific validity is still an owner/consumer gate.

For Paperless GPT activation, --require-compose-profile gpt checks that the parsed, renamed output mapping has exactly COMPOSE_PROFILES=gpt before writing the candidate or replacing .env. This narrow nonsecret option accepts only the currently tested gpt profile; it neither changes the source value nor requires secrets in argv. A mismatch leaves the prior env/permissions/ownership unchanged, or leaves an absent output absent. Do not defer application correctness to a grep after promotion. Without this opt-in constraint, benign empty/omitted inactive core-first values remain allowed.

Traefik's key translation is structural, with missing-source/collision checks:

set -euo pipefail
cd /opt/homelab/services/traefik || exit 1
sudo -n python3 /opt/homelab/scripts/render-sops-env.py \
  --source .env.sops.yaml --output .env \
  --rename CF_DNS_API_TOKEN=CLOUDFLARE_DNS_API_TOKEN \
  --require CLOUDFLARE_DNS_API_TOKEN || exit 1

Serialization and failure boundary¶

The default remains a restricted serializer, not general escaping. Allowed value characters are ASCII letters/digits and _ . / : @ % + , = ! ? ~ - (the spaces in this list are separators, not allowed characters). Dollar signs, either quote, backslash, #, all whitespace/newlines, control characters and non-ASCII values are rejected by default. Do not rewrite or rotate an existing credential just to make it pass: stop for review of a properly tested serializer/application contract. --url-safe KEY still restricts URL-embedded values to RFC 3986 unreserved characters (letters/digits, -._~); dotenv correctness is not URI encoding. Phoenix retains --url-safe POSTGRES_PASSWORD; Komodo and both Firefly modes retain their existing contracts.

Narrow LiteLLM exception: only its callers opt in with --literal POSTGRES_PASSWORD and --url-encode POSTGRES_PASSWORD=POSTGRES_PASSWORD_URLENCODED, while retaining --require POSTGRES_PASSWORD --require LITELLM_MASTER_KEY. The literal value is double-quoted, escaping backslashes and double quotes and doubling dollar signs so Compose env_file and interpolation preserve the raw password. Controls and newlines remain forbidden; required-value and strict placeholder validation still apply. Other keys do not inherit this exception: a dollar sign in a provider key still fails validation even when a literal database password containing it passes. The URL copy is derived with urllib.parse.quote(raw, safe=''), not form encoding (spaces become %20, not +), and existing percent sequences are not decoded (a literal %20 becomes %2520). The result is Compose-specific dotenv, not a shell script or plain docker run --env-file input. Config JSON can double dollar signs for serialization; it is not proof of the actual runtime value. Do not add the derived key to SOPS: an existing POSTGRES_PASSWORD_URLENCODED source key is a collision and fails closed.

LiteLLM's database receives raw POSTGRES_PASSWORD; its proxy URL requires the generated POSTGRES_PASSWORD_URLENCODED. Regenerate old .env files from SOPS before deployment: missing derived values fail closed. This is a representation change, not password rotation or ALTER ROLE. Before live cutover, verify exact byte/string equivalence to the existing working credential through both Compose paths and the decoded URL, and verify database authentication without printing secrets. The authorized host-side source cleanup removes only the unused ANTHROPIC_API_KEY placeholder, decrypting/editing in memory and re-encrypting on the host with all remaining plaintext values unchanged. No validation bypass is added. Follow the LiteLLM rendering and cutover contract; this exception does not authorize other services to opt in.

The helper creates a unique 0700 .decrypted~render-* directory beside the output, writes its candidate .env as 0600, flushes it, and atomically replaces the destination only after validation. Existing broad-mode output becomes 0600 on success; decryption, validation and pre-promotion write failures leave its old bytes and mode unchanged. Destination ownership becomes the rendering identity. Catchable HUP/INT/TERM, validation, decryption, write and rename errors clean up private staging. A signal or cleanup failure after the atomic replace can report failure with the new complete file already present; stop and assess before deploying. An uncatchable kill or power loss can leave a private directory: the existing **/.decrypted~* Git ignore rule protects it even then. Owner-inspect and remove leftovers without printing them before retry; ignored is not encrypted or backup-excluded. Atomic rename is not a power-loss durability or application-consistency certificate.

Firefly retains its service-specific validation in both modes. The manual restore recipe explicitly selects --restricted for shared conversion; missing Python 3.10+/helper blocks before any SOPS call. Automatic migration is deferred: unchanged Komodo/Ansible invocations select the existing shell/SOPS conversion, with ignored private atomic staging and no new Python dependency. The default path does not gain the shared serializer's restricted-character guarantee; its broader escaping/client acceptance remains pending. Do not automatically add --restricted to those consumers without establishing a supported runtime and synthetic consumer compatibility. Current Periphery Python availability is unknown, not claimed absent. Komodo remains a template renderer, already atomic before this repair; it now uses ignored private staging and the restricted secret-value validation, and sets requested ownership before promotion. See its identity contract.

Source tests versus live gates¶

Offline WS2 tests use fake SOPS/credentials only. Whole-procedure tests exercise failed working-directory changes, pulls, missing helper/key and decryption or conversion failures before deployment/state replacement; extracting a renderer alone is not proof that its surrounding procedure is safe. Paperless's synthetic core-first/GPT-later tests check subsequent ordinary Compose calls, not only a single invocation with a profile flag. Firefly's default automatic path is tested without a discoverable python3 or shared helper; this fixture is not evidence about Python's actual presence in Periphery. Local Docker Compose v5.6.1 config tests compare synthetic values through both interpolation and env_file; no daemon, images, pulls, real env or service operation is used. Unsupported characters under the default contract are tested as rejections, not claimed round-trips. LiteLLM's opt-in requires separate synthetic literal/URL consumer checks; the legacy restricted-subset tests alone do not prove that path. These checks do not establish compatibility with another target/client version, real key access, consumer permissions, deployment, credential validity or isolated recovery. Those remain separately approved gates. If Compose is unavailable, the optional client test explicitly skips and client compatibility stays deferred; do not treat the unit serializer checks alone as Compose proof.

File conventions¶

Path pattern	Purpose
`secrets/ansible/*.sops.yaml`	Ansible vault replacements
`secrets/services/<id>/*.sops.yaml`	Per-service credentials
`secrets/appliances/<id>/*.sops.yaml`	Appliance configs (e.g., Saltbox)
`services/<id>/.env.sops.yaml`	Compose stack environment variables
`infra/ansible/inventory/group_vars/all/backup.sops.yaml`	Restic backup credentials (B2, restic password)
`infra/ansible/inventory/group_vars/tailscale/tailscale.sops.yaml`	Ansible `tailscale` role: `tailscale_auth_key` with only `tag:server` (see Phase 7 §5-pre-a).
`secrets/tailscale/nas.sops.yaml`	Manual Synology: `tailscale_auth_key` with only `tag:nas` (Phase 7 §5-pre-b).
`secrets/tailscale/customer-app.sops.yaml`	Optional legacy key: `tailscale_auth_key` with only `tag:customer-app`. Whrrr VMM guests use Ansible `tag:server` key instead.

Secrets inventory¶

File	Keys	Consumer
`infra/ansible/inventory/group_vars/tailscale/tailscale.sops.yaml`	`tailscale_auth_key`	Ansible `tailscale` role (`tag:server` only — Phase 7 §5-pre-a)
`secrets/tailscale/nas.sops.yaml`	`tailscale_auth_key`	Manual whrrr (`tag:nas` only — Phase 7 §5-pre-b)
`secrets/tailscale/customer-app.sops.yaml`	`tailscale_auth_key`	Optional legacy `tag:customer-app` key — Whrrr VMs use Ansible key
`services/traefik/.env.sops.yaml`	`CF_DNS_API_TOKEN`	Traefik (Cloudflare DNS-01 cert resolver)
`services/monitoring/.env.sops.yaml`	`DISCORD_WEBHOOK_URL`, `GF_SECURITY_ADMIN_PASSWORD`, `INCIDENT_ADAPTER_INGEST_TOKEN`	Alertmanager receivers, adapter authentication, Grafana admin login
`services/oneuptime-incident-adapter/.env.sops.yaml`	Database, ingest/admin tokens, project API key	Private adapter persistence, ingestion, audited replay, and OneUptime projection
`services/authentik-outpost/.env.sops.yaml`	`AUTHENTIK_TOKEN`	Authentik proxy outpost on infra-services (ADR-002)
`services/komodo/secrets.sops.yaml`	`KOMODO_DATABASE_PASSWORD`, `KOMODO_INIT_ADMIN_`, `KOMODO_WEBHOOK_SECRET`, `KOMODO_JWT_SECRET`, `KOMODO_OIDC_CLIENT_`	Komodo — rendered to gitignored `compose.env` via `scripts/render-komodo-compose-env.sh`
`services/wazuh/secrets.sops.yaml`	`WAZUH_INDEXER_PASSWORD`, `WAZUH_API_PASSWORD`, `WAZUH_DASHBOARD_PASSWORD`	Retired with the Wazuh stack. Do not render `compose.env` or redeploy.
`services/rexit/.env.sops.yaml`	`REXIT_BEARER_TOKEN`, `HARBOR_USERNAME`, `HARBOR_PASSWORD`	Rexit exporter and ephemeral Harbor pull login on infra-services
`services/rexit/studio.env.sops.yaml`	`REXIT_STUDIO_PASSWORD`	Rexit owner login and destructive-action unlock
`infra/ansible/inventory/group_vars/all/backup.sops.yaml`	`backup_restic_password`, `backup_b2_*`	Ansible `backup-client` role (`sops -d` at apply time, same pattern as `tailscale` SOPS)

Saltbox integration¶

Saltbox expects two config files on disk:

File	Source (encrypted)	Destination on `saltierpoop`
`accounts.yml`	`secrets/appliances/saltierpoop/accounts.sops.yaml`	`/srv/git/saltbox/accounts.yml`
`settings.yml`	`secrets/appliances/saltierpoop/settings.sops.yaml`	`/srv/git/saltbox/settings.yml`

These are decrypted by Ansible during the saltbox-host role (Phase 3) and placed at the paths Saltbox expects. The role uses sops -d and writes the plaintext to the target host. The plaintext never exists in git.

CI integration¶

GitHub Actions uses a dedicated age key (separate from the operator key) so that CI can validate encrypted files without having access to the operator's 1Password.

GitHub Secret	Value
`SOPS_AGE_KEY`	The CI age private key (full content of the key file)

The CI workflow sets SOPS_AGE_KEY as an environment variable. SOPS reads it automatically (no key file needed).

Pre-commit hook¶

The sops-encryption-check hook in .pre-commit-config.yaml verifies that all *.sops.yaml files contain the sops metadata key — catching accidental commits of plaintext files that should have been encrypted.