Operational alert schema

This contract applies to Prometheus rules, Loki ruler alerts, GitHub Actions events, and other sources that enter the incident platform.

Existing Prometheus rules migrate incrementally. CI records their current deficiencies in a baseline and rejects new deficiencies.

Required labels

Label	Purpose	Example
severity	Operator urgency	critical
service	Stable affected service	github-runners
component	Subsystem within the service	runner-controller
environment	Deployment boundary	homelab
owner	Accountable team or owner	homelab
category	Operational failure class	availability
notify	Delivery lane	page or ticket

In the current Alertmanager tree, notify=page is required to select Discord. notify=ticket expresses intent for the OneUptime / operator inbox lane; it does not create a route. Only explicit service/severity/alert-name allowlists select the incident adapter, and unmatched alerts reach blackhole. Grafana Alert Triage visibility is not proof of notification delivery. continue: true evaluates later sibling routes, not receiver-health-based failover. See the current receiver matrix. Do not encode the delivery lane only in severity: some criticals stay tickets (for example GitHub Actions queue), and some pages are capacity criticals on OS volumes.

Every alert must also retain at least one stable target or domain dimension from the expression, source payload, or explicit labels. Common dimensions are instance, host, cluster, repository, runner_group, storage_pool, and backup_target.

Labels are machine fields. Do not put prose, timestamps, generated URLs, or secret values in labels.

Optional labels

• host: canonical inventory host ID when it differs from instance;
• cluster: stable cluster or fleet ID;
• repository: owner/repository for repository-scoped automation;
• workflow: stable workflow file or logical workflow name;
• runner_group: stable runner placement group;
• storage_pool: stable storage pool ID;
• backup_target: stable backup destination ID;
• backup_service: manifest-backed restic job ID when service=backup;
• restore_service: tier-1 repository selected by the latest restore drill;
• secret: allowlisted secret identifier or role, never a secret value;
• inhibit_key: shared failure mode for warning/critical threshold pairs;
• capacity_role: os (boot/system volumes that may page) or bulk (media/content mounts that are ticket-only);
• correlation_key: durable incident correlation path for the OneUptime adapter;
• source: source system when the event did not originate in Prometheus;
• schema_version: event contract version for non-Prometheus adapters.

Severity

Allowed values are:

• info: retained or routed to low-priority channels; not sent to the incident platform by default;
• warning: actionable degradation that needs planned operator attention;
• critical: active outage, data-protection risk, security impact, or imminent service loss requiring prompt intervention.

Do not encode severity in the alert name. Warning and critical threshold pairs use separate rules when their for duration or behavior differs, and share an inhibit_key.

Required annotations

Annotation	Purpose
summary	One-sentence operator-facing symptom
description	Evidence, threshold, duration, and likely impact
dashboard_url	Best Grafana investigation surface
runbook_url	Version-controlled response procedure

Required annotations must be non-empty. URLs must use HTTPS except for explicitly internal test fixtures.

Optional annotations

Annotation	Purpose
logs_url	Scoped Grafana Explore or log dashboard link
action_url	Bounded diagnostic/remediation entry point
investigation_url	Other authenticated investigation surface
prometheus_url	Prometheus expression or rule link

action_url must not encode a secret or execute a destructive action on GET. It should open an authenticated approval or dispatch surface.

Naming conventions

Services and components

• Use lowercase kebab-case.
• Use a durable operational capability, not a container name that may change.
• Examples: github-runners, backup, monitoring, storage.
• Components refine the service: runner-controller, restic, alertmanager, postgresql.

Hosts and instances

• host uses the immutable inventory ID, such as infra-services.
• instance is the scrape/source identity. Prefer the inventory ID for node-level metrics instead of an IP address.
• Do not rename hosts to describe a temporary role.

Ownership

owner is a stable, routable identifier. The default for lab infrastructure is homelab. Do not use a person's display name or chat handle.

Categories

Use one of these initial categories:

• availability;
• capacity;
• performance;
• data-protection;
• security;
• configuration;
• automation;
• dependency.

Adding a category requires updating this contract and its linter tests.

Fingerprints and correlation

Prometheus/Alertmanager's per-alert fingerprint is the alert identity. Do not replace it with a hash of formatted text.

Incident correlation uses a versioned ordered tuple:

v1/environment/service/component/category/<domain-dimension>

Examples:

v1/homelab/github-runners/runner-controller/availability/linux
v1/homelab/storage/filesystem/capacity/primary
v1/homelab/backup/restic/data-protection/infra-services/monitoring
v1/homelab/plex/server/availability/media-host-01

The domain mapping is version controlled with the OneUptime workflow. Changing the tuple requires a new version so active incidents do not silently merge or split.

Alertmanager groupKey and alert fingerprint are retained in the timeline for traceability.

Firing and recovery events

• The first firing event creates the alert and, if needed, its incident.
• Repeated firing with the same fingerprint updates last_seen, evidence, and duration; it does not create another alert.
• A resolved event updates the same fingerprint.
• Partial group resolution leaves the incident open.
• The incident resolves only when all attached fingerprints are resolved.
• A later firing event after full resolution reopens the same operational correlation or creates a linked recurrence, according to the versioned OneUptime workflow policy.

Alertmanager receivers for the incident platform must set send_resolved: true.

Non-Prometheus sources

GitHub Actions and custom checks should use this canonical source envelope as a translation contract. It is not a directly accepted API payload. The current private /v1/alertmanager endpoint accepts an AlertmanagerWebhook, not this shape; this document does not provide or claim an implemented generic translator.

The example intentionally selects notify: ticket, consistent with runner alert policy. Critical severity does not imply paging, and the lane alone does not guarantee a receiver:

schema_version: v1
source: github-actions
source_event_id: "repository/run-id/job-id"
status: firing
starts_at: "2026-07-29T20:00:00Z"
ends_at: null
labels:
  severity: critical
  notify: ticket
  service: github-runners
  component: runner-controller
  environment: homelab
  instance: runner-host-01
  owner: homelab
  category: availability
  runner_group: linux
annotations:
  summary: "No eligible Linux runner is online"
  description: "All Linux x64 runners have been unavailable for ten minutes."
  dashboard_url: "https://grafana.infra.realemail.app/d/gha-fleet-overview"
  logs_url: "https://grafana.infra.realemail.app/d/gha-runner-logs"
  runbook_url: "https://github.com/spadoople/homelab/blob/main/docs/runbooks/github-actions-observability.md"
  action_url: "https://github.com/spadoople/homelab/actions/workflows/runner-fleet-dispatch.yml"

A future source translator must derive a deterministic fingerprint from source, source_event_id, and the stable affected-object identity, excluding status, timestamps, summaries and descriptions. Resolved events must reuse the same source_event_id and fingerprint. The current adapter consumes the supplied Alertmanager fingerprint; it does not compute this source-envelope mapping.

Translation and acceptance boundary

Before enabling any non-Prometheus producer, review and test its translator:

• map source semantics to an explicitly approved alert name, service, component, category, stable domain labels and matching correlation_key; do not invent an allowlisted alert name merely to bypass validation;
• retain the intentional notify lane and verify receiver selection separately from severity; direct submission to the adapter does not traverse Alertmanager or deliver a Discord page;
• construct the Alertmanager webhook fields (receiver, version, groupKey, status, and exactly one member of alerts with status, labels, annotations, startsAt, endsAt, and fingerprint), rather than POSTing the envelope above. Both top-level status and alerts[0].status are required (firing or resolved); alerts[0].status must equal the top-level status; neither omitting per-alert status nor sending mismatched statuses is accepted;
• convert snake-case timestamps and firing ends_at: null to the supported Alertmanager representation, preserving identity across firing and resolution;
• do not blindly copy source, source_event_id or schema_version into labels: the current parser rejects unsupported labels and enforces domain, correlation, annotation and URL allowlists in addition to the webhook shape;
• retain private-network and bearer-token ingress restrictions. Independently accept repeat, firing, recovery, correlation and intended delivery behavior.

Conforming to this abstract envelope is source-contract validation, not proof of an implemented integration or successful delivery. Any new route, source or allowlist expansion requires separate review; this example authorizes none.

Validation policy

scripts/lint-alert-schema.py validates Prometheus rule files, not generic event ingestion. The offline tests/test_ws4_monitoring_contracts.py also parses this fenced example against the shared required-label/annotation contract. The service-local services/oneuptime-incident-adapter/tests/test_ws4_envelope_boundary.py tests the actual adapter's rejection of the untranslated envelope using the same fixture. These checks send no events. The baseline permits known legacy omissions but has exact-deficiency semantics:

• a new rule must satisfy the complete contract;
• a legacy rule may not acquire a new omission;
• fixing a legacy omission requires updating the baseline;
• deleting or renaming a baseline rule requires updating the baseline;
• duplicate alert names and unsupported severity/category values fail.

Run:

uv run python scripts/lint-alert-schema.py
uv run python -m unittest tests.test_alert_schema -v