Skip to content

Alert triage

Use this runbook when a Discord alert needs more context than the notification body can carry. Alertmanager is the workflow source of truth, Grafana is the first investigation surface, and the alert annotations link to the most relevant dashboard and runbook for the alert class.

Entry points

Surface Use
Alertmanager Active groups, silences, repeats, and resolved state
Alert Triage All firing alerts from Prometheus ALERTS series
Node Overview Host CPU, memory, disk, network, and uptime
Docker Overview infra-services container health from cAdvisor
Backup / Restic Triage Backup job status, repo freshness, and restore-test state
Coordinated OS Patching Patch metrics and reboot state

Scope

Prometheus node alerts cover the generated node targets in monitoring/targets/nodes.yml: graylog, harbor-registry, infra-services, octoprint, phoenix, prox, pulse, recordurbate, saltierpoop, and ubuncap.

Backup drilldowns start with hosts running the backup-client role through the docker_hosts group: infra-services and phoenix. Saltbox containers on saltierpoop, inventory-only appliances, retired guests, and customer app lifecycle internals are not homelab-owned alert producers unless inventory and generator scope changes.

Standard workflow

  1. Open the Discord link to the alert dashboard or runbook.
  2. In Alertmanager, confirm whether the alert is still firing, grouped, inhibited, or silenced.
  3. In Grafana Alert Triage, filter by alertname, instance, component, and severity.
  4. Use the alert-specific dashboard to inspect raw metrics and trend context.
  5. Check the matching runbook section below before making host changes.
  6. After remediation, verify Prometheus resolves the alert and Alertmanager sends the resolved notification if configured for that receiver.

Alert classes

Node and infra capacity

Alerts: HostDown, DiskSpaceLow, DiskSpaceCritical, HighMemoryUsage, HighCPUUsage, InfraServicesDiskPressure, InfraServicesDiskCritical, and InfraServicesMemoryLow.

Use Node Overview first. For infra-services disk or memory pressure, continue to infra-services capacity and resize, because that host has tighter thresholds than generic node alerts.

Useful Prometheus checks:

up{job="node", instance="<host>"}
1 - (node_filesystem_avail_bytes{instance="<host>", fstype=~"ext4|xfs|btrfs"}
  / node_filesystem_size_bytes{instance="<host>", fstype=~"ext4|xfs|btrfs"})
1 - (node_memory_MemAvailable_bytes{instance="<host>"}
  / node_memory_MemTotal_bytes{instance="<host>"})

Containers

Alerts: ContainerRestarting and ContainerMissing.

Use Docker Overview first, then check the service's services/<name>/README.md or stack directory. Promtail collects infra-services Docker logs, so Grafana Explore can search Loki by compose_project, compose_service, or container labels when the failing workload runs on infra-services.

Patching

Alerts: OSPatchStale, OSPatchOrchestrateFailed, and PatchRebootPending.

Use the patching dashboard and ARA link from the alert. The remediation path is Coordinated OS patching.

Backups and restore tests

Alerts: ResticMetricsMissing, ResticMetricsStale, ResticExpectedBackupMissing, ResticBackupRunFailed, ResticBackupTimerInactive, ResticBackupStale, ResticBackupFailed, RestoreTestStale, and RestoreTestFailed.

Use backup alert triage. Do not delete or relocate backup artifacts while triaging without explicit owner approval.

TLS

Alert: TLSCertExpiringSoon.

Use the alert table to identify cn, then check Traefik ACME state and public edge runbook context in Public edge incidents. The source metric is traefik_tls_certs_not_after_timestamp.

Ansible convergence

Alert: AnsiblePullStale.

The source metric is the ansible_pull.prom textfile mtime. Check ansible-pull-apply.service, ansible-pull-check.service, and recent ARA runs for the target host.

Komodo drift

Alerts: KomodoSecretDrift and KomodoSecretDriftCheckStale.

Use Komodo GitHub webhook relay, especially the webhook signature and Mongo password drift sections. These alerts usually mean the rendered secret and live consumer are out of sync, not that SOPS rendering itself failed.

Validation after changes

Run these checks after changing alert rules, routing, dashboards, or runbooks:

promtool check rules monitoring/prometheus/alerts/*.yml
amtool check-config monitoring/alertmanager/alertmanager.yml
uv run python scripts/lint-markdown.py --fix
uv run python scripts/lint-markdown.py

For dashboard JSON, parse each changed dashboard with Python:

python -m json.tool monitoring/grafana/dashboards/alert-triage.json > /dev/null
python -m json.tool monitoring/grafana/dashboards/backup-triage.json > /dev/null