Operational alert schema

This contract applies to Prometheus rules, Loki ruler alerts, GitHub Actions events, and other sources that enter the incident platform.

Existing Prometheus rules migrate incrementally. CI records their current deficiencies in a baseline and rejects new deficiencies.

Required labels

Label	Purpose	Example
severity	Operator urgency	critical
service	Stable affected service	github-runners
component	Subsystem within the service	runner-controller
environment	Deployment boundary	homelab
owner	Accountable team or owner	homelab
category	Operational failure class	availability

Every alert must also retain at least one stable target or domain dimension from the expression, source payload, or explicit labels. Common dimensions are instance, host, cluster, repository, runner_group, storage_pool, and backup_target.

Labels are machine fields. Do not put prose, timestamps, generated URLs, or secret values in labels.

Optional labels

• host: canonical inventory host ID when it differs from instance;
• cluster: stable cluster or fleet ID;
• repository: owner/repository for repository-scoped automation;
• workflow: stable workflow file or logical workflow name;
• runner_group: stable runner placement group;
• storage_pool: stable storage pool ID;
• backup_target: stable backup destination ID;
• inhibit_key: shared failure mode for warning/critical threshold pairs;
• source: source system when the event did not originate in Prometheus;
• schema_version: event contract version for non-Prometheus adapters.

Severity

Allowed values are:

• info: retained or routed to low-priority channels; not sent to the incident platform by default;
• warning: actionable degradation that needs planned operator attention;
• critical: active outage, data-protection risk, security impact, or imminent service loss requiring prompt intervention.

Do not encode severity in the alert name. Warning and critical threshold pairs use separate rules when their for duration or behavior differs, and share an inhibit_key.

Required annotations

Annotation	Purpose
summary	One-sentence operator-facing symptom
description	Evidence, threshold, duration, and likely impact
dashboard_url	Best Grafana investigation surface
runbook_url	Version-controlled response procedure

Required annotations must be non-empty. URLs must use HTTPS except for explicitly internal test fixtures.

Optional annotations

Annotation	Purpose
logs_url	Scoped Grafana Explore or log dashboard link
action_url	Bounded diagnostic/remediation entry point
investigation_url	Other authenticated investigation surface
prometheus_url	Prometheus expression or rule link

action_url must not encode a secret or execute a destructive action on GET. It should open an authenticated approval or dispatch surface.

Naming conventions

Services and components

• Use lowercase kebab-case.
• Use a durable operational capability, not a container name that may change.
• Examples: github-runners, backup, monitoring, storage.
• Components refine the service: runner-controller, restic, alertmanager, postgresql.

Hosts and instances

• host uses the immutable inventory ID, such as infra-services.
• instance is the scrape/source identity. Prefer the inventory ID for node-level metrics instead of an IP address.
• Do not rename hosts to describe a temporary role.

Ownership

owner is a stable, routable identifier. The default for lab infrastructure is homelab. Do not use a person's display name or chat handle.

Categories

Use one of these initial categories:

• availability;
• capacity;
• performance;
• data-protection;
• security;
• configuration;
• automation;
• dependency.

Adding a category requires updating this contract and its linter tests.

Fingerprints and correlation

Prometheus/Alertmanager's per-alert fingerprint is the alert identity. Do not replace it with a hash of formatted text.

Incident correlation uses a versioned ordered tuple:

v1/environment/service/component/category/<domain-dimension>

Examples:

v1/homelab/github-runners/runner-controller/availability/linux
v1/homelab/storage/filesystem/capacity/primary
v1/homelab/backup/restic/data-protection/synology-primary
v1/homelab/plex/server/availability/media-host-01

The domain mapping is version controlled with the OneUptime workflow. Changing the tuple requires a new version so active incidents do not silently merge or split.

Alertmanager groupKey and alert fingerprint are retained in the timeline for traceability.

Firing and recovery events

• The first firing event creates the alert and, if needed, its incident.
• Repeated firing with the same fingerprint updates last_seen, evidence, and duration; it does not create another alert.
• A resolved event updates the same fingerprint.
• Partial group resolution leaves the incident open.
• The incident resolves only when all attached fingerprints are resolved.
• A later firing event after full resolution reopens the same operational correlation or creates a linked recurrence, according to the versioned OneUptime workflow policy.

Alertmanager receivers for the incident platform must set send_resolved: true.

Non-Prometheus sources

GitHub Actions and custom checks must map their payload into this envelope:

schema_version: v1
source: github-actions
source_event_id: "repository/run-id/job-id"
status: firing
starts_at: "2026-07-29T20:00:00Z"
ends_at: null
labels:
  severity: critical
  service: github-runners
  component: runner-controller
  environment: homelab
  instance: runner-host-01
  owner: homelab
  category: availability
  runner_group: linux
annotations:
  summary: "No eligible Linux runner is online"
  description: "All Linux x64 runners have been unavailable for ten minutes."
  dashboard_url: "https://grafana.infra.realemail.app/d/gha-fleet-overview"
  logs_url: "https://grafana.infra.realemail.app/d/gha-runner-logs"
  runbook_url: "https://github.com/spadoople/homelab/blob/main/docs/runbooks/github-actions-observability.md"
  action_url: "https://github.com/spadoople/homelab/actions/workflows/runner-fleet-dispatch.yml"

The adapter fingerprint is a deterministic hash of source, source_event_id, and the stable affected-object identity. It must not include status, timestamps, summaries, or descriptions.

Resolved events reuse the same source_event_id and fingerprint.

Validation policy

scripts/lint-alert-schema.py validates Prometheus rule files. The baseline permits known legacy omissions but has exact-deficiency semantics:

• a new rule must satisfy the complete contract;
• a legacy rule may not acquire a new omission;
• fixing a legacy omission requires updating the baseline;
• deleting or renaming a baseline rule requires updating the baseline;
• duplicate alert names and unsupported severity/category values fail.

Run:

uv run python scripts/lint-alert-schema.py
uv run python -m unittest tests.test_alert_schema -v