Alert triage¶
Use this runbook when an alert needs more context than the notification body can
carry. Allowlisted domains create OneUptime incidents through the incident
adapter. Discord receives only notify=page alerts (page-worthy criticals);
notify=ticket warnings and demoted criticals stay in OneUptime and Grafana.
ntfy remains the critical OS-patching side channel.
Entry points¶
| Surface | Use |
|---|---|
| Alertmanager | Active groups, silences, repeats, and resolved state |
| OneUptime | Durable incidents for allowlisted domains (Authentik + local login) |
| Alert Triage | All firing alerts from Prometheus ALERTS series |
| Node Overview | Host CPU, memory, disk, network, and uptime |
| Docker Overview | infra-services container health from cAdvisor |
| Backup / Restic Triage | Backup job status, repo freshness, and restore-test state |
| Coordinated OS Patching | Patch metrics and reboot state |
| Monitoring dead-man | External Healthchecks.io heartbeat when Prom/AM cannot notify |
Notification lanes¶
| Label | Destination |
|---|---|
notify=page |
Discord (+ OneUptime when the alertname is adapter-allowlisted) |
notify=ticket |
OneUptime when allowlisted; otherwise Grafana/Alertmanager only |
severity=critical + service=os-patching |
ntfy (unchanged side channel) |
Scope¶
Prometheus node alerts cover the generated node targets in
monitoring/targets/nodes.yml: graylog, harbor-registry,
infra-services, octoprint, phoenix, prox, pulse, recordurbate,
saltierpoop, and ubuncap.
Backup drilldowns start with hosts running the backup-client role through the
docker_hosts group: infra-services and phoenix. Saltbox containers on
saltierpoop, inventory-only appliances, retired guests, and customer app
lifecycle internals are not homelab-owned alert producers unless inventory and
generator scope changes.
Standard workflow¶
- Open the Discord or OneUptime link to the alert dashboard or runbook.
- In Alertmanager, confirm whether the alert is still firing, grouped, inhibited, or silenced.
- In Grafana Alert Triage, filter by
alertname,instance,component, andseverity. - Use the alert-specific dashboard to inspect raw metrics and trend context.
- Check the matching runbook section below before making host changes.
- After remediation, verify Prometheus resolves the alert and Alertmanager sends the resolved notification if configured for that receiver.
Alert classes¶
Node and infra capacity¶
Alerts: NodeExporterUnavailable, NodeFilesystemLow,
NodeFilesystemCritical, NodeFilesystemBulkPressure, NodeMemoryLow,
NodeCPUHigh, NodeDiskLatencyHigh, NodeOOMKillDetected,
InfraServicesDiskPressure, InfraServicesDiskCritical, and
InfraServicesMemoryLow.
Capacity policy:
capacity_role |
Mounts | Behavior |
|---|---|---|
os |
Boot/system volumes (everything not listed as bulk) | Warning + critical; critical uses notify=page |
bulk |
saltierpoop /mnt/local/Media/*, recordurbate /mnt/streams, ubuncap /mnt/prawn |
NodeFilesystemBulkPressure warning + notify=ticket only — never Discord |
Use Node Overview first. For infra-services disk or memory pressure, continue to infra-services capacity and resize, because that host has tighter thresholds than generic node alerts.
Useful Prometheus checks:
up{job="node", instance="<host>"}
1 - (node_filesystem_avail_bytes{instance="<host>", fstype=~"ext4|xfs|btrfs"}
/ node_filesystem_size_bytes{instance="<host>", fstype=~"ext4|xfs|btrfs"})
1 - (node_memory_MemAvailable_bytes{instance="<host>"}
/ node_memory_MemTotal_bytes{instance="<host>"})
For NodeDiskLatencyHigh, compare the alert with iostat -xmd 1 5 on the
affected machine. The rule measures average read/write completion time from
the matching node exporter counters and excludes Proxmox LXCs, whose
/proc/diskstats exposes the hypervisor's physical devices.
Large jumps close to the host's uptime in
node_disk_io_time_weighted_seconds_total are not device latency. Linux block
layer regression 691b052139c9 can corrupt weighted disk statistics; update
to a kernel containing that fix and reboot. Check NVMe SMART health and kernel
I/O errors separately before treating the alert as hardware failure.
Containers¶
Alerts: ContainerRestarting and ContainerMissing.
Use Docker Overview first, then check the service's services/<name>/README.md
or stack directory. Alloy collects infra-services Docker logs, so Grafana
Explore can search Loki by compose_project, compose_service, or container
labels when the failing workload runs on infra-services.
Patching¶
Alerts: OSPatchStale, OSPatchOrchestrateFailed, and
PatchRebootPending.
Use the patching dashboard and ARA link from the alert. The remediation path is Coordinated OS patching.
For PatchRebootPending specifically: the alert follows
homelab_patch_reboot_required in patch.prom. After converge, boot and
ansible-pull sync that gauge from /var/run/reboot-required. If it still stays
1 after a successful reboot, triage (and manual sync): PatchRebootPending
triage.
Backups and restore tests¶
Alerts: ResticMetricsMissing, ResticMetricsStale,
ResticExpectedBackupMissing, ResticBackupRunFailed,
ResticBackupTimerInactive, ResticBackupStale, ResticBackupFailed,
RestoreTestMetricsMissing, RestoreTestStale, and RestoreTestFailed.
Use backup alert triage. Do not delete or relocate backup artifacts while triaging without explicit owner approval.
TLS¶
Alert: TLSCertExpiringSoon.
Correlate by Traefik certificate cn (volatile sans/serial labels are
aggregated away). Use the firing alert to identify cn, then check Traefik ACME
state and public edge runbook context in Public edge incidents.
The source metric is traefik_tls_certs_not_after.
Monitoring platform¶
Alerts: CentralAlloyUnavailable and LokiIngestionErrors.
Correlate per infra-services host. Alloy means the central metrics/logs gateway
scrape is down. Loki means samples are being discarded (limits, disk, or push
path). Host root-disk pressure remains under InfraServicesDisk* —
the old PrometheusStoragePressure duplicate was removed.
Ansible convergence¶
Alert: AnsiblePullStale.
The source metric is the ansible_pull.prom textfile mtime. Check
ansible-pull-apply.service, ansible-pull-check.service, and recent ARA runs
for the target host.
Komodo drift¶
Alerts: KomodoSecretDrift and KomodoSecretDriftCheckStale.
Use Komodo GitHub webhook relay, especially the webhook signature and Mongo password drift sections. These alerts usually mean the rendered secret and live consumer are out of sync, not that SOPS rendering itself failed.
Validation after changes¶
Run these checks after changing alert rules, routing, dashboards, or runbooks:
promtool check rules monitoring/prometheus/alerts/*.yml
promtool test rules monitoring/prometheus/tests/*.yml
uv run python scripts/lint-alert-schema.py
# Validate the rendered Alertmanager config, not the source placeholders.
amtool check-config /tmp/alertmanager.yml
uv run python scripts/lint-markdown.py --fix
uv run python scripts/lint-markdown.py
For dashboard JSON, parse each changed dashboard with Python: