TL;DR
Discord was drowning in ~868 notifications / 96h, mostly chronic filesystem
capacity and invalid OneUptime adapter webhooks. Alertmanager now pages Discord
only on notify=page; bulk/media mounts are ticket-only; the adapter no longer
stalls reconcile on space-containing runner correlation keys.
Phase status
| Area |
Status |
Discord notify=page routing |
Shipped (PR) |
OS vs bulk capacity_role |
Shipped |
| Adapter dead-letter / 400 spam |
Fixed live + in PR |
| sonarqube node_exporter |
Repaired live |
| ansible-pull stale guests |
Repaired live; OMZ force: true in PR |
| pulse root disk |
Cleaned live (~88% → ~74%) |
| prox mem/latency |
Evidence only — mem real, latency tolerable |
What shipped
- Alert schema requires
notify: page|ticket; filesystem rules use
capacity_role: os|bulk (alert-schema.md,
alert-triage.md).
- Alertmanager default
blackhole; Discord matcher notify="page"; expanded
OneUptime adapter routes for filesystem + monitoring plane alerts.
- Adapter: slugify correlation segments, flexible path regex, skip invalid
allowlisted actives during reconcile, accept
notify/capacity_role labels.
- Fortify: OMZ plugin git
force: true so customer-app ansible-pull converges.
Known gaps / drift
- Hosts pull
origin/main — merge this PR so ansible-pull stops reverting live
overlays.
- prox swap is exhausted; treat
NodeMemoryLow as real capacity pressure (ticket).
- GitHub hosted-style runner names still emit Inactive tickets until fleet
hygiene removes them.
What remains
| Item |
Owner |
Priority |
| Merge alert-noise PR; rebuild adapter + recreate Alertmanager on main |
Owner |
P0 |
| Decide prox RAM expansion vs tolerate swap pressure |
Owner |
P2 |
| Stage 7/8 OneUptime notification soak (orthogonal) |
Owner |
P2 |
flowchart LR
Prom[Prometheus_Loki] --> AM[Alertmanager]
AM -->|"notify=page"| Discord[Discord]
AM -->|"allowlisted ticket/page"| OU[OneUptime_adapter]
AM -->|"critical os-patching"| Ntfy[ntfy]
AM -->|default| BH[blackhole]