OneUptime runner incident pilot¶

This runbook evaluates durable OneUptime incidents for four standardized GitHub runner alert shapes. The deployed child route is initially synthetic-only: real runner alerts continue to Discord but cannot enter the native canvas. The unrelated critical-patching ntfy route is unchanged.

Recorded result¶

Status: rejected on 2026-07-31. Keep the workflow disabled.

The sequential create and repeat phases passed, but the required concurrent create phase produced two incidents for one correlation key. OneUptime accepted both webhook requests, both workflow runs completed through Log Created, and no native uniqueness or transaction boundary prevented the duplicate.

Retained evidence:

acceptance run ID: pilot-20260731T073400Z;
successful primary incident: 3403ae6e-23d7-4929-a4e0-b026975a9c63;
concurrent workflow runs: 5dd191d8-31f4-4928-9b47-43e1fb451bd6 and 3c87ef89-fbe3-4905-9cbc-358c1345bf49;
duplicate incidents: 0edc3107-ace1-44d1-8413-05d873d78dfd and 4703491b-0e6a-4373-807d-4f83c7d6d42c.

The workflow was disabled after the failed gate. Do not delete the workflow runs or duplicate incidents, and do not continue the later lifecycle phases. The Alertmanager route retains synthetic="true"; real runner alerts remain on the existing direct notification path. Production correlation now requires the race-safe adapter specified below and in ADR-003.

OneUptime 11.7.3 accepts a webhook before the asynchronous canvas finishes and can execute multiple workflow runs concurrently. Its native Find/Create/Update components provide no correlation lock, compare-and-swap, or failed-run replay. The canvas therefore cannot become the authoritative incident path unless the concurrent tests prove behavior and a durable reconciliation design closes the accepted-but-failed delivery gap. The expected production solution is the narrow race-safe adapter allowed by ADR-003.

Scope¶

The shadow route accepts only alerts matching all four constraints:

service="github-runners"
severity=~"critical|warning"
alertname=~"GitHubRunner(Offline|ServiceStopped|NoIdleCapacity|Inactive)"
synthetic="true"

Prometheus-generated runner alerts do not carry synthetic="true" and cannot match this route. Remove that safety matcher only in the adapter promotion PR, after reconciliation and concurrency gates pass.

Alertmanager reaches oneuptime-ingress:7849 over the private oneuptime_application Docker network. The workflow webhook path is a secret; do not route machine delivery through Authentik or expose a broad bypass.

The OneUptime child route uses group_by: ["..."], which disables aggregation for this receiver. The workflow processes exactly one Alertmanager fingerprint per execution; OneUptime correlates separate executions by correlation_key. Do not change the route back to correlation-key grouping unless the workflow has first been redesigned to iterate over multiple entries in alerts[].

The pilot alerts are:

GitHubRunnerOffline;
GitHubRunnerServiceStopped;
GitHubRunnerNoIdleCapacity;
GitHubRunnerInactive.

GitHubRunnerOffline and GitHubRunnerServiceStopped intentionally share a correlation key for the same runner. This proves that resolving one fingerprint does not close an incident while the other remains active.

Before you begin¶

Complete these checks before editing the workflow:

Confirm the deployed oneuptime-app image is the repository-pinned oneuptime/app:11.7.3 build. These component names and variable semantics were verified against 11.7.3 and must be re-audited after an upgrade.
Confirm the capacity gate and isolated backup/restore drill in the operations platform migration passed. Do not make OneUptime an incident dependency without those earlier stages.
Confirm oneuptime-app, oneuptime-ingress, oneuptime-postgres, oneuptime-redis, and oneuptime-clickhouse are healthy on infra-services.
Sign in to https://oneuptime.infra.realemail.app and select the homelab operations project.
Use a Project Owner or Project Admin account. Creating incident custom fields and editing workflows requires project-level permissions.
Keep the workflow disabled until the repository change is merged, the monitoring stack is redeployed, and the pre-enable checks pass.
Use a synthetic run ID that has never been used before. Reusing an old ID can match an incident retained from an earlier acceptance run.

This runbook distinguishes two test paths:

direct workflow tests post controlled Alertmanager-shaped JSON from the Alertmanager container directly to the private OneUptime ingress. They prove canvas behavior, duplicate handling, correlation, and resolution without creating a Prometheus alert;
route tests add a synthetic alert through Alertmanager with amtool. They prove matcher selection, private-network delivery, continue: true, Discord delivery, and send_resolved: true.

The direct tests intentionally bypass Alertmanager routing. Passing them does not prove the route; both test paths are required.

Create the disabled workflow¶

The custom fields used by this workflow are project-level incident fields. They are not created inside the workflow editor. Creating a field defines a place that every incident in this OneUptime project can use; the workflow will populate that place when it creates or updates an incident.

Create the workflow shell¶

In the homelab operations project:

Open Workflows from the main product navigation.
Select Create Workflow.
Set Name to Alertmanager Runner Pilot.
Set Description to Correlates the additive GitHub runner Alertmanager pilot into incidents.
Confirm Enabled is off.
Save the workflow and open its Builder tab.
Add the Webhook trigger from the trigger palette.
In the trigger modal, set Identifier to Alertmanager and click Save.
Reopen the saved Webhook trigger if necessary and copy its generated URL.
Open the workflow's Overview page. In Workflow Details, confirm Enabled is Disabled. If it is enabled, select Edit, switch Enabled off, and save.

Keep the webhook URL in a password manager or another temporary secret location. Do not add it to a workflow note, incident, issue, or Git file. If the URL is ever printed in a terminal, chat, screenshot, unencrypted diff, or workflow log:

open the workflow's Settings page;
under Webhook Secret Key, select Reset Secret Key and confirm;
return to the Builder, reopen the Webhook trigger, and copy the new URL;
replace the encrypted SOPS value before continuing.

Create the incident custom fields¶

Leave the workflow editor temporarily; the fields are created elsewhere:

Confirm that the project selector still shows the homelab operations project where the workflow was created.
Open Incidents from OneUptime's main product navigation.
In the Incidents sidebar, find the collapsed Settings section and expand it.
Select Custom Fields under that Settings section. Do not select an individual incident's Custom Fields tab, and do not use the similarly named custom-field settings under Alerts or Monitors.
Confirm that the page heading says Incident Custom Fields.

The direct route has this form if the collapsed sidebar is difficult to find:

https://oneuptime.infra.realemail.app/dashboard/<project-id>/incidents/settings/custom-fields

Keep the existing project ID from the URL after /dashboard/; do not literally enter <project-id>.

Create the first field using the page's create button:

Name: Correlation Key
Description: Stable Alertmanager correlation key used to find and update the same incident.
Custom Field Type: Text
Dropdown Options: leave empty; this input should not appear for a Text field.

Save it, then create the second field:

Name: Active Alert Fingerprints
Description: JSON array of active Alertmanager fingerprints for this correlated incident.
Custom Field Type: Text
Dropdown Options: leave empty.

Save it, then create the third field:

Name: Last Seen At
Description: UTC time when the workflow last received a firing event for this incident.
Custom Field Type: Text
Dropdown Options: leave empty.

Use those names exactly, including capitalization and spaces. The workflow field picker displays these human-readable names; OneUptime does not ask for a separate machine name on this screen.

After saving both definitions:

Confirm that all three rows appear on the Incident Custom Fields page and all show a Text field type.
Return to Workflows and reopen Alertmanager Runner Pilot.
Keep the workflow disabled. Creating the field definitions does not require enabling or testing the webhook.

If Custom Fields or the create button is absent, verify that the signed-in user is a Project Owner or Project Admin for the selected project. Do not work around a missing incident field by creating an Alert or Monitor custom field; those are different resources and will not appear on an incident action.

Field semantics¶

Treat the incoming correlation_key label as the incident identity. Store it unchanged in Correlation Key. Treat each Alertmanager alerts[].fingerprint as an independently active alert identity and store the deduplicated set in Active Alert Fingerprints as a compact JSON array, for example:

["4a09d1c0f1a2b3c4","9702a430d5e6f7a8"]

Do not store a comma-separated list. JSON preserves an unambiguous empty value ([]) and can be parsed safely by later workflow steps.

Set Last Seen At on every firing delivery, including a repeated fingerprint. Preserve Alertmanager's groupKey, event time, workflow-observed time, and computed duration in the incident evidence so a repeated firing updates the durable record rather than becoming an invisible no-op.

Build the workflow canvas¶

Keep the workflow disabled throughout this section. Open the workflow's Builder tab and retain the existing Alertmanager Webhook trigger.

OneUptime uses Identifier for the operator-visible name of a canvas block. Use every identifier in this runbook exactly; workflow variable references use it verbatim, including spaces and capitalization.

Each block has two separate save behaviors:

click Save in the block configuration modal after changing its Identifier, code, arguments, or fields;
wait for the Builder's saved indicator after adding or changing connection lines.

Closing a block modal with Cancel discards its field changes even though the canvas itself may say it is saved.

The completed canvas has this shape:

flowchart LR
    W[Alertmanager webhook] --> N[Normalize Alertmanager Payload]
    N --> F[Find Many Incidents]
    N -. Error .-> NE[Log Normalize Error]
    F --> D[Decide Incident Transition]
    F -. Error .-> FE[Log Find Error]
    D --> C{Action is create?}
    D -. Error .-> DE[Log Decision Error]
    C -->|Yes| CI[Create One Incident]
    C -->|No| X{Action is noop?}
    X -->|Yes| NL[Log No-op]
    X -->|No| UI[Update One Incident]
    CI --> CL[Log Created]
    CI -. Error .-> CE[Log Create Error]
    UI --> UL[Log Updated]
    UI -. Error .-> UE[Log Update Error]

Every component with an Error output must connect that output to a dedicated Log component. For each error Log:

add Log from the Utils category;
set the exact Identifier and static Value specified below;
click Save;
connect the source component's Error port to the Log's In port.

Do not put the request body, webhook path, or an entire component return object in an error Log. Component failures are already recorded in the workflow run; the static Log value identifies which error path executed without duplicating payload data.

Add Normalize Alertmanager Payload¶

Search the component palette for Run Custom JavaScript.
Add it to the right of the Webhook trigger.
Set Identifier to Normalize Alertmanager Payload.
Connect the Webhook trigger's Out port to this component's In port.
Arguments is a JSON code editor, not a property form. Paste the complete object below into it:

{
  "contentLength": "{{local.components.Alertmanager.returnValues.request-headers.content-length}}",
  "payload": {
    "status": "{{local.components.Alertmanager.returnValues.request-body.status}}",
    "alerts": [
      {
        "status": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].status}}",
        "labels": {
          "alertname": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].labels.alertname}}",
          "correlation_key": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].labels.correlation_key}}",
          "service": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].labels.service}}",
          "severity": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].labels.severity}}",
          "synthetic": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].labels.synthetic}}"
        },
        "annotations": {
          "summary": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].annotations.summary}}",
          "description": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].annotations.description}}",
          "dashboard_url": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].annotations.dashboard_url}}",
          "runbook_url": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].annotations.runbook_url}}"
        },
        "startsAt": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].startsAt}}",
        "endsAt": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].endsAt}}",
        "generatorURL": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].generatorURL}}",
        "fingerprint": "{{local.components.Alertmanager.returnValues.request-body.alerts[0].fingerprint}}"
      }
    ],
    "groupKey": "{{local.components.Alertmanager.returnValues.request-body.groupKey}}"
  }
}

The references deliberately select scalar leaves instead of putting the whole request-body reference inside a quoted JSON value. OneUptime 11.7.3 stringifies an object before interpolation but does not escape backslashes already present in that JSON text. Alertmanager's normal groupKey contains quoted matchers, so whole-object interpolation produces invalid component arguments before the JavaScript runs.

The synthetic helper always supplies every selected field. Do not add optional leaf references unless the helper also supplies that field on every event; OneUptime leaves a missing reference as literal {{...}} text.

If constructing the object with the picker instead of pasting it, keep every reference inside its surrounding JSON string quotes. Use Webhook (Alertmanager) and descend through Request Body or Request Headers to the exact leaf shown above. Confirm that labels.fingerprint is absent: the fingerprint belongs to the alert object, not its labels.

Paste the following into JavaScript Code:

const parse = (value) => {
  if (typeof value === "string") {
    return JSON.parse(value);
  }
  return value;
};

const contentLength = Number(args.contentLength || 0);
if (
  !Number.isFinite(contentLength) ||
  contentLength < 1 ||
  contentLength > 262144
) {
  throw new Error("Alertmanager payload exceeds the 256 KiB pilot limit");
}

const payloadText =
  typeof args.payload === "string"
    ? args.payload
    : JSON.stringify(args.payload || {});
if (payloadText.length > 262144) {
  throw new Error("Alertmanager payload exceeds the 256 KiB pilot limit");
}

const payload = parse(payloadText);
if (!payload || !Array.isArray(payload.alerts)) {
  throw new Error("Alertmanager payload does not contain alerts[]");
}
if (payload.alerts.length !== 1) {
  throw new Error(
    `Expected exactly one grouped alert; received ${payload.alerts.length}`,
  );
}

const alert = payload.alerts[0];
const labels = alert.labels || {};
const annotations = alert.annotations || {};
const status = String(alert.status || payload.status || "").toLowerCase();

if (!["firing", "resolved"].includes(status)) {
  throw new Error(`Unsupported Alertmanager status: ${status}`);
}

const required = {
  alertname: labels.alertname,
  correlationKey: labels.correlation_key,
  fingerprint: alert.fingerprint,
  severity: labels.severity,
  service: labels.service,
  synthetic: labels.synthetic,
  groupKey: payload.groupKey,
};
for (const [name, value] of Object.entries(required)) {
  if (!String(value || "").trim()) {
    throw new Error(`Alertmanager alert is missing ${name}`);
  }
}

const allowedAlerts = new Set([
  "GitHubRunnerOffline",
  "GitHubRunnerServiceStopped",
  "GitHubRunnerNoIdleCapacity",
  "GitHubRunnerInactive",
]);
if (!allowedAlerts.has(String(required.alertname))) {
  throw new Error(`Unsupported alertname: ${required.alertname}`);
}
if (String(required.service) !== "github-runners") {
  throw new Error(`Unsupported service: ${required.service}`);
}
if (String(required.synthetic) !== "true") {
  throw new Error("Shadow workflow accepts only synthetic=true");
}
const severity = String(required.severity).toLowerCase();
if (!["critical", "warning"].includes(severity)) {
  throw new Error(`Unsupported severity: ${required.severity}`);
}
const correlationKey = String(required.correlationKey);
if (
  !/^v1\/homelab\/github-runners\/runner-controller\/(availability|capacity|automation)\/\S+$/.test(
    correlationKey,
  )
) {
  throw new Error(`Unsupported correlation key: ${correlationKey}`);
}
const correlationParts = correlationKey.split("/");
const incidentTitle = `[RUNNER] ${correlationParts[4]}: ${correlationParts
  .slice(5)
  .join("/")}`.slice(0, 500);
if (String(required.fingerprint).length > 200) {
  throw new Error("Alertmanager fingerprint exceeds 200 characters");
}

const startsAt = String(alert.startsAt || "");
const endsAt = String(alert.endsAt || "");
if (!startsAt || Number.isNaN(Date.parse(startsAt))) {
  throw new Error(`Invalid startsAt: ${startsAt}`);
}
if (
  status === "resolved" &&
  (!endsAt || Number.isNaN(Date.parse(endsAt)))
) {
  throw new Error(`Invalid resolved endsAt: ${endsAt}`);
}

const links = [
  annotations.dashboard_url,
  annotations.logs_url,
  annotations.runbook_url,
  annotations.investigation_url,
  annotations.prometheus_url,
].filter(Boolean);

return {
  status,
  alertname: String(required.alertname),
  correlationKey,
  incidentTitle,
  fingerprint: String(required.fingerprint),
  severity,
  service: String(required.service),
  groupKey: String(required.groupKey),
  summary: String(annotations.summary || required.alertname),
  description: String(annotations.description || ""),
  startsAt,
  endsAt,
  observedAt: new Date().toISOString(),
  generatorURL: String(alert.generatorURL || ""),
  links,
};

Click Save.
Add an error Log with:
Identifier: Log Normalize Error;
Value: ERROR: Normalize Alertmanager Payload failed; inspect this workflow run.
Connect Normalize Alertmanager Payload → Error to Log Normalize Error → In.

Add Find Many Incidents¶

Search for Find Many Incidents and add it after the normalize component.
Set Identifier to Find Many Incidents.
Connect Normalize Alertmanager Payload → Success to Find Many Incidents → In.
Set Query to:

{
  "title": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.incidentTitle}}"
}

Set Select Fields to:

{
  "_id": true,
  "createdAt": true,
  "title": true,
  "description": true,
  "currentIncidentStateId": true,
  "incidentSeverityId": true,
  "customFields": true
}

Set Skip to 0 and Limit to 3.
Click Save.
Add an error Log with:
Identifier: Log Find Error;
Value: ERROR: Find Many Incidents failed; inspect this workflow run.
Connect Find Many Incidents → Error to Log Find Error → In.

OneUptime's workflow query editor cannot safely query a key inside the JSON custom-field column. The normalize component therefore derives a deterministic title from the correlation key, and Find Many uses exact title equality to bound its result. The Decision component still verifies the Correlation Key custom field before treating a row as correlated.

The limit is 3 so the next component can detect an unexpected third recurrence without interpolating it. Three matching rows is a stop condition; the native canvas is not a durable incident index.

Add Decide Incident Transition¶

Add another Run Custom JavaScript component.
Set Identifier to Decide Incident Transition.
Connect Find Many Incidents → Success to this component's In.
Paste this complete object into Arguments:

{
  "normalized": {
    "status": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.status}}",
    "alertname": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.alertname}}",
    "correlationKey": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.correlationKey}}",
    "incidentTitle": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.incidentTitle}}",
    "fingerprint": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.fingerprint}}",
    "severity": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.severity}}",
    "service": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.service}}",
    "groupKey": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.groupKey}}",
    "summary": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.summary}}",
    "description": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.description}}",
    "startsAt": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.startsAt}}",
    "endsAt": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.endsAt}}",
    "observedAt": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.observedAt}}",
    "generatorURL": "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.generatorURL}}",
    "links": [
      "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.links[0]}}",
      "{{local.components.Normalize Alertmanager Payload.returnValues.returnValue.links[1]}}"
    ]
  },
  "incidentCount": "{{local.components.Find Many Incidents.returnValues.models.length}}",
  "incidents": [
    {
      "_id": "{{local.components.Find Many Incidents.returnValues.models[0]._id}}",
      "createdAt": "{{local.components.Find Many Incidents.returnValues.models[0].createdAt}}",
      "description": "{{local.components.Find Many Incidents.returnValues.models[0].description}}",
      "currentIncidentStateId": "{{local.components.Find Many Incidents.returnValues.models[0].currentIncidentStateId}}",
      "incidentSeverityId": "{{local.components.Find Many Incidents.returnValues.models[0].incidentSeverityId}}",
      "customFields": {
        "Correlation Key": "{{local.components.Find Many Incidents.returnValues.models[0].customFields.Correlation Key}}",
        "Active Alert Fingerprints": "{{local.components.Find Many Incidents.returnValues.models[0].customFields.Active Alert Fingerprints}}",
        "Last Seen At": "{{local.components.Find Many Incidents.returnValues.models[0].customFields.Last Seen At}}"
      }
    },
    {
      "_id": "{{local.components.Find Many Incidents.returnValues.models[1]._id}}",
      "createdAt": "{{local.components.Find Many Incidents.returnValues.models[1].createdAt}}",
      "description": "{{local.components.Find Many Incidents.returnValues.models[1].description}}",
      "currentIncidentStateId": "{{local.components.Find Many Incidents.returnValues.models[1].currentIncidentStateId}}",
      "incidentSeverityId": "{{local.components.Find Many Incidents.returnValues.models[1].incidentSeverityId}}",
      "customFields": {
        "Correlation Key": "{{local.components.Find Many Incidents.returnValues.models[1].customFields.Correlation Key}}",
        "Active Alert Fingerprints": "{{local.components.Find Many Incidents.returnValues.models[1].customFields.Active Alert Fingerprints}}",
        "Last Seen At": "{{local.components.Find Many Incidents.returnValues.models[1].customFields.Last Seen At}}"
      }
    }
  ]
}

As with the first JavaScript component, every reference selects a scalar leaf. Do not replace either reconstructed object with a quoted whole-object reference. Missing models[0] or models[1] fields remain literal template text; the JavaScript below removes those placeholder rows before correlation.

Paste the following into JavaScript Code:

const parse = (value, fallback) => {
  if (value === null || value === undefined || value === "") {
    return fallback;
  }
  if (typeof value === "string") {
    return JSON.parse(value);
  }
  return value;
};

const IDENTIFIED_STATE_ID = "669adc5a-6021-44fa-b18b-43d54b52dc09";
const RESOLVED_STATE_ID = "590e33d6-3ba8-44f5-98ed-0f87bf485e0b";
const CRITICAL_SEVERITY_ID = "ee0180b3-48fe-4303-a758-df0c3d260736";
const MAJOR_SEVERITY_ID = "f76fe1ee-177c-4789-a66d-6efbcadf53be";

const normalized = parse(args.normalized, {});
const incidentCount = Number(args.incidentCount);
const incidentCandidates = parse(args.incidents, []);
if (!Array.isArray(incidentCandidates)) {
  throw new Error("Find Many Incidents did not return an array");
}
const incidents = incidentCandidates.filter(
  (candidate) =>
    candidate._id && !String(candidate._id).startsWith("{{local.components."),
);
if (
  !Number.isInteger(incidentCount) ||
  incidentCount < 0 ||
  incidentCount > 2 ||
  incidents.length !== incidentCount
) {
  throw new Error(
    `Expected at most two reconstructed incidents; received ${incidentCount}`,
  );
}

const correlatedIncidents = incidents.filter((candidate) => {
  const fields = candidate.customFields || {};
  return fields["Correlation Key"] === normalized.correlationKey;
});
const openIncidents = correlatedIncidents.filter(
  (candidate) => candidate.currentIncidentStateId !== RESOLVED_STATE_ID,
);
if (openIncidents.length > 1) {
  throw new Error(
    `Found ${openIncidents.length} open incidents for one correlation key`,
  );
}
const incident = openIncidents[0];
const priorResolvedIncident = correlatedIncidents
  .filter((candidate) => candidate.currentIncidentStateId === RESOLVED_STATE_ID)
  .sort(
    (left, right) =>
      Date.parse(String(right.createdAt || "")) -
      Date.parse(String(left.createdAt || "")),
  )[0];

if (!incident && normalized.status === "resolved") {
  return {
    action: "noop",
    reason: "resolution arrived without a matching open incident",
  };
}

const existingFields = incident ? incident.customFields || {} : {};
let activeFingerprints = parse(
  existingFields["Active Alert Fingerprints"],
  [],
);
if (!Array.isArray(activeFingerprints)) {
  throw new Error("Active Alert Fingerprints is not a JSON array");
}
activeFingerprints = [...new Set(activeFingerprints.map(String))].sort();

const wasActive = activeFingerprints.includes(normalized.fingerprint);
let nextFingerprints = [...activeFingerprints];
if (normalized.status === "firing" && !wasActive) {
  nextFingerprints.push(normalized.fingerprint);
}
if (normalized.status === "resolved" && wasActive) {
  nextFingerprints = nextFingerprints.filter(
    (fingerprint) => fingerprint !== normalized.fingerprint,
  );
}
nextFingerprints.sort();

if (incident && normalized.status === "resolved" && !wasActive) {
  return {
    action: "noop",
    reason: "resolved fingerprint was not active",
  };
}

const eventAt =
  normalized.status === "resolved"
    ? normalized.endsAt || normalized.startsAt
    : normalized.startsAt;
const durationSeconds = Math.max(
  0,
  Math.floor(
    (Date.parse(normalized.observedAt) - Date.parse(normalized.startsAt)) / 1000,
  ),
);
const marker =
  `<!-- alertmanager:${normalized.fingerprint}:` +
  `${normalized.status}:${normalized.observedAt} -->`;
const evidence = [
  marker,
  `### Alertmanager ${normalized.status}${wasActive ? " repeat" : ""}: ${
    normalized.alertname
  }`,
  `- Event time: ${eventAt}`,
  `- Observed by workflow: ${normalized.observedAt}`,
  `- Firing duration at observation: ${durationSeconds} seconds`,
  `- Fingerprint: \`${normalized.fingerprint}\``,
  `- Correlation key: \`${normalized.correlationKey}\``,
  `- Alertmanager group key: \`${normalized.groupKey}\``,
  !incident && priorResolvedIncident
    ? `- Prior resolved recurrence: incident \`${priorResolvedIncident._id}\``
    : "",
  normalized.description ? `- Evidence: ${normalized.description}` : "",
  normalized.generatorURL
    ? `- Prometheus source: ${normalized.generatorURL}`
    : "",
  ...normalized.links.map((link) => `- Related: ${link}`),
]
  .filter(Boolean)
  .join("\n");

const previousDescription = incident ? String(incident.description || "") : "";
const description = previousDescription.includes(marker)
  ? previousDescription
  : [previousDescription, evidence].filter(Boolean).join("\n\n");
const customFields = {
  ...existingFields,
  "Correlation Key": normalized.correlationKey,
  "Active Alert Fingerprints": JSON.stringify(nextFingerprints),
  ...(normalized.status === "firing"
    ? { "Last Seen At": normalized.observedAt }
    : {}),
};

if (!incident) {
  const createData = {
    title: normalized.incidentTitle,
    description,
    currentIncidentStateId: IDENTIFIED_STATE_ID,
    incidentSeverityId:
      normalized.severity === "critical"
        ? CRITICAL_SEVERITY_ID
        : MAJOR_SEVERITY_ID,
    customFields,
  };
  return {
    action: "create",
    createDataJson: JSON.stringify(createData),
  };
}

const shouldResolve =
  normalized.status === "resolved" && nextFingerprints.length === 0;
const updateData = {
  description,
  customFields,
  ...(shouldResolve
    ? { currentIncidentStateId: RESOLVED_STATE_ID }
    : {}),
};

return {
  action: shouldResolve ? "resolve" : "update",
  incidentId: incident._id,
  queryJson: JSON.stringify({ _id: incident._id }),
  updateDataJson: JSON.stringify(updateData),
};

The UUIDs above are the live IDs verified in the homelab operations project. If an Incident State or Incident Severity is deleted and recreated, stop and update these constants before enabling the workflow.

Immediately before enabling, verify the IDs from PostgreSQL without displaying database credentials:

sudo docker exec -i oneuptime-postgres sh -c \
  'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At' <<'SQL'
SELECT '"'"'state|'"'"' || "_id" || '"'"'|'"'"' || "name"
FROM "IncidentState"
WHERE "_id" IN (
  '"'"'669adc5a-6021-44fa-b18b-43d54b52dc09'"'"',
  '"'"'590e33d6-3ba8-44f5-98ed-0f87bf485e0b'"'"'
);
SELECT '"'"'severity|'"'"' || "_id" || '"'"'|'"'"' || "name"
FROM "IncidentSeverity"
WHERE "_id" IN (
  '"'"'ee0180b3-48fe-4303-a758-df0c3d260736'"'"',
  '"'"'f76fe1ee-177c-4789-a66d-6efbcadf53be'"'"'
);
SQL

Require exactly these mappings:

state|669adc5a-6021-44fa-b18b-43d54b52dc09|Identified
state|590e33d6-3ba8-44f5-98ed-0f87bf485e0b|Resolved
severity|ee0180b3-48fe-4303-a758-df0c3d260736|Critical Incident
severity|f76fe1ee-177c-4789-a66d-6efbcadf53be|Major Incident

Click Save.
Add an error Log with:
Identifier: Log Decision Error;
Value: ERROR: Decide Incident Transition failed; inspect this workflow run.
Connect Decide Incident Transition → Error to Log Decision Error → In.

Add the create branch¶

Add an If / Else component and set Identifier to Action is Create.
Connect Decide Incident Transition → Success to its In port.
Configure:
Input 1 Type: Text;
Input 1: click component, select Run Custom JavaScript (Decide Incident Transition), select Value, and submit the picker. It inserts a reference ending in returnValue}}. Put the cursor immediately before the final }} and type .action;
Operator: Equal To;
Input 2 Type: Text;
Input 2: create.

The final Input 1 must be exactly:

{{local.components.Decide Incident Transition.returnValues.returnValue.action}}

Click Save.
Add Create One Incident and set its Identifier to Create One Incident.
Connect Action is Create → Yes to its In port.
In JSON Object, click component, select Run Custom JavaScript (Decide Incident Transition), select Value, and submit the picker.
Put the cursor immediately before the final }} and type .createDataJson. The final value must be exactly:

{{local.components.Decide Incident Transition.returnValues.returnValue.createDataJson}}

Click Save.
Add a success Log with:
Identifier: Log Created;
Value: Runner incident created.
Connect Create One Incident → Success to Log Created → In.
Add an error Log with:
Identifier: Log Create Error;
Value: ERROR: Create One Incident failed; inspect this workflow run.
Connect Create One Incident → Error to Log Create Error → In.

Add the update and no-op branches¶

Add a second If / Else component and set Identifier to Action is No-op.
Connect Action is Create → No to its In port.
Configure it exactly like Action is Create, including the .action suffix, but set Input 2 to noop.
Confirm Input 1 is:

{{local.components.Decide Incident Transition.returnValues.returnValue.action}}

Click Save.
Add a Log with Identifier Log No-op.
In its Value, use the component picker for Decide Incident Transition → Value, then append .reason immediately before the final }}. The final value must be:

{{local.components.Decide Incident Transition.returnValues.returnValue.reason}}

Click Save.
Connect Action is No-op → Yes to Log No-op → In.
Add Update One Incident and set its Identifier to Update One Incident.
Connect Action is No-op → No to its In port.
In Query, use the picker for Decide Incident Transition → Value, then append .queryJson before the final }}. Confirm:

{{local.components.Decide Incident Transition.returnValues.returnValue.queryJson}}

In Data (JSON Object), use the same picker and append .updateDataJson. Confirm:

{{local.components.Decide Incident Transition.returnValues.returnValue.updateDataJson}}

Click Save.
Add a success Log with:
Identifier: Log Updated;
Value: Runner incident updated or resolved.
Connect Update One Incident → Success to Log Updated → In.
Add an error Log with:
Identifier: Log Update Error;
Value: ERROR: Update One Incident failed; inspect this workflow run.
Connect Update One Incident → Error to Log Update Error → In.

The firing path now creates an incident only when no matching open incident exists, adds fingerprints without duplicates, and appends evidence. The resolved path removes only the matching fingerprint and changes the incident to Resolved only after the active set becomes empty.

The version-1 recurrence policy does not reopen a resolved incident. A later firing creates a new incident with the same correlation key and records the most recent resolved incident ID in the new evidence section.

OneUptime's incident service adds a feed item when an incident is created, when its description is updated, and when its state changes. The accumulated description is the durable event evidence; the generated feed entries make those updates visible in the incident activity timeline. A stale resolution for a fingerprint that is not active takes the no-op branch and does not add timeline noise. Repeated firing deliberately updates last-seen evidence.

Update One Incident uses OneUptime's IncidentService, not a hookless SQL update. In 11.7.3, an update containing currentIncidentStateId invokes changeIncidentState from onUpdateSuccess, which writes the IncidentStateTimeline and state-change feed. Acceptance must still verify the Resolved timeline entry; a changed row alone is not sufficient evidence.

Inspect before enabling¶

Wait for OneUptime to show that the canvas has saved, then verify:

every non-Log component has its expected incoming connection;
every component with an Error port has an error Log connection;
neither condition contains a manually guessed component ID;
Create One Incident receives createDataJson;
Update One Incident receives both queryJson and updateDataJson;
the workflow remains disabled.

If concurrent duplicate runs create more than one incident, stop the pilot. Native workflow primitives have failed the idempotency gate; implement the narrow adapter allowed by ADR-003 instead of accepting duplicate state.

Store the webhook path in SOPS¶

Do this in the feature-branch development clone before merge, not by leaving a dirty secret file in /opt/homelab on the production host.

Copy only the generated /workflow/trigger/<secret-key> path. Discard the scheme and host. A query string or fragment is not valid and is rejected because the Compose renderer substitutes this value into Alertmanager YAML. Copy it directly from the OneUptime UI or password manager into the SOPS editor; do not pass the full URL to echo, a parser command, or shell argv.
Open the tracked encrypted file with SOPS in an environment that has an authorized operator age key. Do not copy the host automation key out of /etc/homelab/age-key.txt:

sops services/monitoring/.env.sops.yaml

Set the top-level value:

ONEUPTIME_ALERTMANAGER_WEBHOOK_PATH: "/workflow/trigger/opaque-value"

Save and exit SOPS.
Verify that the file is still encrypted:

pre-commit run sops-encryption-check \
  --files services/monitoring/.env.sops.yaml

Review only the encrypted representation:

git diff --no-textconv -- services/monitoring/.env.sops.yaml

This repository has a SOPS text-conversion diff driver. A normal git diff on this file may decrypt values for display. Always use --no-textconv when reviewing a SOPS diff, and never paste that value into a commit message or PR description.

The renderer rejects an empty value, a full URL, and a path that does not begin with /workflow/trigger/. Keep the workflow disabled after committing and pushing the encrypted change.

Pre-merge canvas inspection¶

Wait for OneUptime's saved indicator, then count and inspect the graph:

16 nodes: one trigger, two JavaScript components, one Find component, two conditions, one Create component, one Update component, and eight Logs;
15 connections, including every success, branch, and error path shown in this runbook;
no unconfigured default identifiers such as log-1 or incident-create-one-1;
Action is Create → Input 1 ends in .action;
Create One Incident → JSON Object ends in .createDataJson;
Action is No-op → Input 1 ends in .action;
Log No-op → Value ends in .reason;
Update One Incident → Query ends in .queryJson;
Update One Incident → Data (JSON Object) ends in .updateDataJson;
all error Logs contain static messages rather than request or return data;
the workflow Enabled setting remains off.

Do not merge the route while any item above is incomplete.

Native-canvas safety boundary¶

The canvas error branches are diagnostics, not delivery recovery. OneUptime returns a successful webhook acknowledgement when it schedules the run; a later JavaScript, query, create, or update failure cannot make Alertmanager retry that event. A lost resolved event can therefore leave a stale incident.

The native workflow is also a read-modify-write sequence. OneUptime 11.7.3 can run workflow jobs concurrently and does not provide a unique correlation constraint or atomic fingerprint-set update. A successful race drill is useful evidence but is not a general proof that all future schedules are serialized.

For those reasons:

keep synthetic="true" on the Alertmanager route;
treat any workflow error, duplicate incident, lost fingerprint, stale resolution, or ambiguous 100-row lookup as a stop condition;
do not describe the native canvas as the canonical incident path;
retain workflow runs and incidents when a gate fails.

The adapter promotion must provide:

a bounded, source-restricted, rate-limited private endpoint;
payload schema and allowlist validation;
durable enqueue before returning success to Alertmanager;
a unique indexed correlation identity;
transactional or compare-and-swap fingerprint updates;
idempotent firing and resolution event storage;
dead-letter visibility and payload-safe replay;
reconciliation between active Alertmanager alerts and open incidents;
metrics and alerts for rejected, failed, delayed, replayed, and reconciled events.

Only the adapter promotion PR may remove the synthetic matcher.

Merge and deploy the route¶

The repository PR must be green and the owner must merge it before these host commands run. Merging places the encrypted path, renderer validation, Alertmanager route, private network attachment, and acceptance script on main.

After infra-services has pulled the merge through Komodo or git pull --ff-only, run:

cd /opt/homelab
sudo bash scripts/render-monitoring-secrets.sh
sudo docker compose \
  --project-directory services/monitoring \
  --env-file services/monitoring/.env \
  config --quiet
sudo docker compose \
  --project-directory services/monitoring \
  --env-file services/monitoring/.env \
  up -d --force-recreate alertmanager

Validate the recreated container without printing the rendered webhook URL:

sudo docker inspect alertmanager \
  --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}'
sudo docker exec alertmanager \
  amtool check-config /tmp/alertmanager.yml
sudo docker exec alertmanager \
  amtool --alertmanager.url=http://127.0.0.1:9093 config routes show
sudo docker inspect alertmanager \
  --format '{{if index .NetworkSettings.Networks "oneuptime_application"}}attached{{else}}missing{{end}}'
sudo docker exec alertmanager \
  wget -q --spider http://oneuptime-ingress:7849/api/status/live

Expected results:

container status is healthy;
amtool check-config reports success;
the route tree contains oneuptime-runner-pilot, the four shadow matchers, group_by: [...], continue: true, group_wait: 10s, and group_interval: 30s;
alertmanager is attached to oneuptime_application.

Do not run amtool config show in a recorded terminal: the full rendered configuration includes receiver URLs and can expose the webhook path.

Run the general live monitoring checks:

sudo bash scripts/validate-monitoring.sh --live

Fix any failure before enabling the workflow.

Pre-enable checks¶

In OneUptime, reopen Workflows → Alertmanager Runner Pilot → Builder.
Repeat the node, connection, identifier, and variable-reference inspection above.
Open the workflow's Runs & Logs page and note the current last run, if any.
On infra-services, check for real active runner alerts:

sudo docker exec alertmanager \
  amtool --alertmanager.url=http://127.0.0.1:9093 \
  alert query service=github-runners

If any non-synthetic pilot alert is active, stop. Investigate it or wait for it to resolve so acceptance evidence cannot be confused with a real incident.
Open the workflow's Overview page, select Edit in Workflow Details, switch Enabled on, and save.
Return to the workflow list and confirm its status is enabled.

Webhook responses only mean OneUptime scheduled a run. A successful HTTP response is not acceptance evidence; inspect the workflow run and incident after every phase below.

Use these UI paths during acceptance:

workflow execution: Workflows → Alertmanager Runner Pilot → Runs & Logs, then open the newest run and inspect its executed blocks and final Log;
incident state: Incidents → Incidents, search for Synthetic runner pilot, open the candidate, then inspect Details, Custom Fields, and its activity/state history.

The incident-list search may not index custom-field values. Never treat an empty correlation-key search as proof that no incident exists; use the synthetic title and run timestamp, then confirm the exact Correlation Key inside the incident.

Direct workflow acceptance¶

Run the test script on infra-services from /opt/homelab. It reads only the path from the root-only rendered monitoring environment and never prints it. Each phase must exit zero, print OneUptime's trigger acknowledgement, and end with the script's PASS line before UI verification begins.

Create unique identifiers in the same shell:

cd /opt/homelab
PRIMARY_RUN_ID="pilot-$(date -u +%Y%m%dT%H%M%SZ)"
RACE_RUN_ID="${PRIMARY_RUN_ID}-race"
ORDER_RUN_ID="${PRIMARY_RUN_ID}-ordering"
EARLY_RESOLVE_RUN_ID="${PRIMARY_RUN_ID}-early-resolve"

Create one incident¶

sudo scripts/test-oneuptime-runner-pilot.sh \
  offline-fire "${PRIMARY_RUN_ID}"

In OneUptime, refresh Runs & Logs, open View Logs for the newest run, record its Run ID, and verify:

workflow status is Completed and the final block is Log Created;
exactly one incident has correlation key ending in synthetic-${PRIMARY_RUN_ID};
state is Identified and severity is Critical Incident;
Active Alert Fingerprints is a one-element JSON array containing synthetic-offline-${PRIMARY_RUN_ID};
Last Seen At is a valid UTC timestamp from this workflow run;
the description contains the firing evidence, Grafana link, runbook link, fingerprint, and correlation key.

Record the workflow run ID and incident ID before continuing.

Prove repeat idempotency¶

sudo scripts/test-oneuptime-runner-pilot.sh \
  offline-repeat "${PRIMARY_RUN_ID}"

Verify:

the new workflow run ends at Log Updated;
the original incident ID is unchanged;
the project still has exactly one incident with this correlation key;
the active fingerprint array still contains one value;
Last Seen At advances;
a firing-repeat evidence section records the workflow observation time, duration, fingerprint, and Alertmanager group key.

Prove concurrent-create behavior¶

The race test must use the unused RACE_RUN_ID. Running it against the primary incident would test concurrent repeat updates, not concurrent creation.

sudo scripts/test-oneuptime-runner-pilot.sh \
  offline-concurrent "${RACE_RUN_ID}"

Wait for both workflow runs to finish. Query incidents by the correlation key ending in synthetic-${RACE_RUN_ID} and verify exactly one incident exists.

If two incidents exist:

disable the workflow immediately;
do not delete either incident;
record both workflow run IDs and incident IDs;
mark the native workflow idempotency gate failed;
implement the narrow race-safe adapter allowed by ADR-003 before retrying.

If exactly one exists, add the second fingerprint to that race-test incident:

sudo scripts/test-oneuptime-runner-pilot.sh \
  service-fire "${RACE_RUN_ID}"

Verify the same incident now contains both fingerprints. Then submit both resolutions concurrently:

sudo scripts/test-oneuptime-runner-pilot.sh \
  both-resolve-concurrent "${RACE_RUN_ID}"

Verify that the race-test incident moves to Resolved and its active array is []. If it remains open with either stale fingerprint, disable the workflow and fail the race-safety gate using the same evidence procedure above.

Prove out-of-order and mixed-event behavior¶

Send a resolution before any firing event exists:

sudo scripts/test-oneuptime-runner-pilot.sh \
  offline-resolve "${EARLY_RESOLVE_RUN_ID}"

Verify the run is Completed, ends at Log No-op, records resolution arrived without a matching open incident, and creates no incident.

Now submit a chronologically later resolution concurrently with its firing event:

sudo scripts/test-oneuptime-runner-pilot.sh \
  fire-resolve-concurrent "${ORDER_RUN_ID}"

The event timestamps make Resolved the required final state. Verify exactly one incident exists for this correlation, it is Resolved, and its active array is []. An open incident, no incident, duplicates, or a stale fingerprint fails the race-safety gate.

Prove multi-fingerprint correlation¶

Add the second fingerprint to the primary incident:

sudo scripts/test-oneuptime-runner-pilot.sh \
  service-fire "${PRIMARY_RUN_ID}"

Verify:

no second incident was created;
the incident ID is still the original primary incident;
the incident remains Identified;
Active Alert Fingerprints contains exactly the sorted offline and service fingerprints for PRIMARY_RUN_ID;
a service-stopped firing evidence section was appended once.

Resolve only the offline fingerprint:

sudo scripts/test-oneuptime-runner-pilot.sh \
  offline-resolve "${PRIMARY_RUN_ID}"

Verify the same incident remains open and its active array contains only synthetic-service-${PRIMARY_RUN_ID}.

Resolve the final fingerprint:

sudo scripts/test-oneuptime-runner-pilot.sh \
  service-resolve "${PRIMARY_RUN_ID}"

Verify:

the same incident moves to Resolved;
Active Alert Fingerprints is [];
no replacement incident was created;
the description retains all firing and resolution evidence.

Repeat the final resolution:

sudo scripts/test-oneuptime-runner-pilot.sh \
  service-resolve "${PRIMARY_RUN_ID}"

Verify the run ends at Log No-op, no new incident appears, and the resolved incident remains unchanged.

Alertmanager route acceptance¶

This test begins at Alertmanager, not Prometheus. The Prometheus expressions, labels, and deterministic correlation keys are covered separately by monitoring/prometheus/tests/github-runner-pilot.yml and monitoring/prometheus/tests/github-runner-inactive.yml.

Use a third unique identifier:

ROUTE_RUN_ID="${PRIMARY_RUN_ID}-route"
sudo scripts/test-oneuptime-runner-pilot.sh \
  route-fire "${ROUTE_RUN_ID}"

Wait at least 45 seconds, then verify:

Alertmanager accepted the synthetic alert;
OneUptime created exactly one incident whose correlation key ends in synthetic-${ROUTE_RUN_ID};
the OneUptime workflow run is Completed;
Discord received one clearly marked synthetic firing notification;
no ntfy message is expected because ntfy is scoped to critical patching.

Inspect Alertmanager's current view if needed:

sudo scripts/test-oneuptime-runner-pilot.sh \
  route-query "${ROUTE_RUN_ID}"

Resolve through Alertmanager:

sudo scripts/test-oneuptime-runner-pilot.sh \
  route-resolve "${ROUTE_RUN_ID}"

Wait at least 45 seconds and verify the same OneUptime incident resolves. Record whether the existing Discord receiver emits a resolved notification, but do not change that receiver as part of this additive pilot.

Restart persistence¶

After all synthetic incidents are resolved, restart only the stateless OneUptime application and ingress containers:

cd /opt/homelab
sudo docker compose \
  --project-directory services/oneuptime \
  --env-file services/oneuptime/compose.env \
  restart app ingress
sudo bash services/oneuptime/scripts/smoke-test.sh --live

Reopen each incident and verify IDs, state, custom fields, description evidence, and workflow logs remain available. Do not restart PostgreSQL, ClickHouse, or Redis for this gate; database restart and restore behavior belongs to the separate recovery drill.

Record acceptance evidence¶

Add a PR comment or milestone evidence note using this template:

Commit:
Acceptance started/finished (UTC):
Primary run ID:
Race run ID:
Ordering run ID:
Early-resolution run ID:
Route run ID:
Workflow ID:
Primary incident ID:
Race incident ID:
Ordering incident ID:
Route incident ID:
Workflow run IDs by phase:
Observed Discord firing delivery:
Observed Discord resolved behavior:
Concurrent-create incident count:
Restart persistence result:
Operator:
Deviations or failures:

Do not record the webhook URL/path, SOPS ciphertext, unrelated environment values, or request headers.

Rollback¶

Immediate safe stop¶

Open Workflows → Alertmanager Runner Pilot → Overview.
Select Edit in Workflow Details, switch Enabled off, and save.
Confirm the workflow list shows it disabled.
Leave synthetic and real incidents intact as evidence.

The Alertmanager child route may continue attempting the disabled webhook, but continue: true keeps direct Discord routing independent. The patching ntfy route is also independent.

Remove delivery¶

Prepare a focused rollback branch from current main that removes:

the oneuptime-runner-pilot child route and receiver;
Alertmanager's oneuptime-application network attachment;
the Alertmanager startup requirement and substitution for ONEUPTIME_ALERTMANAGER_WEBHOOK_PATH;
the renderer's requirement for that key, if no other consumer needs it.

Merge the rollback, pull it on infra-services, rerender monitoring secrets, and force-recreate Alertmanager with the same deployment commands above. Validate container health, amtool check-config, and direct Discord delivery.

If the webhook was exposed, use Settings → Webhook Secret Key → Reset Secret Key and replace the encrypted SOPS value before any future retry. Do not delete OneUptime incidents or database history during rollback.