> For the complete documentation index, see [llms.txt](https://docs-lunar.earthly.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs-lunar.earthly.dev/configuration/hub-configuration/webhooks.md).

# Outbound Webhooks

Outbound webhooks. Register an HTTP endpoint and the Hub POSTs it a signed JSON notification whenever a subscribed event happens, with retries and a record of every attempt.

Outbound webhooks let another system react to Lunar the moment something happens, instead of polling the SQL API. You register an HTTP endpoint for one or more events; the Hub POSTs a signed JSON notification to it for each one, retries an endpoint that fails, and records every attempt.

Every `lunar webhooks` command needs [`hub.operate`](https://docs-lunar.earthly.dev/docs/authorization.md#hub-operate): a subscription's URL can be a credential, and a delivery holds what its receiver answered. The Hub must have `HUB_SECRETS_ENCRYPTION_KEY` configured: subscription URLs and signing secrets are stored encrypted with it.

## Events

`lunar webhooks events` lists advertised events with the fields of their payloads:

| Event              | Sent When                                                                                                                                                                                     |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ping`             | A subscription is created, and on `lunar webhooks ping`. Every subscription receives it, so its endpoint can be checked at any time                                                           |
| `bypass.exercised` | A change was let through under a bypass: a release approved by `lunar policy ok-release`, or a PR merged, with at least one gating check masked                                               |
| `script_run.error` | A collector, policy or cataloger run failed: the script exited non-zero, or the Hub could not complete the run. Covers the runs the Hub schedules and the collectors the CI tracer runs in CI |

### `bypass.exercised`

The change-management signal: a [bypass](https://docs-lunar.earthly.dev/docs/lunar-cli.md#lunar-policy-bypass-release) was actually used to let a change through. It is sent once per consumption of a gate, never for a bypass that was only created:

* **Release gate**: `lunar policy ok-release` approved a commit with at least one release-blocking check masked by a bypass. A refused release lets nothing through and sends nothing, even when a bypass masked some of its checks.
* **PR gate**: a pull or merge request merged with gating checks that a bypass masked, or that were still pending and a commit pin let through.

A repeat of the same consumption, such as a CI re-run calling `ok-release` again for the same commit, or a redelivered merge webhook, sends nothing new. A re-run that masks a different set of checks, for example because a flaky check now passes, is a new consumption and sends a new event.

```json
{
  "id": "c3e1...",
  "event": "bypass.exercised",
  "version": 1,
  "created_at": "2026-10-08T10:12:03Z",
  "data": {
    "component": "github.com/acme/api",
    "domain": "payments",
    "gate": "release",
    "sha": "9c1e...",
    "pr_number": 412,
    "exercised_at": "2026-10-08T10:12:03Z",
    "checks": [
      {
        "name": "sbom.no-critical-vulns",
        "enforcement": "block-pr-and-release",
        "bypass": {
          "id": "5a0c...",
          "policy": "sbom",
          "reason": "hotfix for INC-231",
          "actor": "jdoe",
          "verified_role": "github:maintain",
          "source": "cli",
          "self_bypass": false,
          "created_at": "2026-10-08T09:40:00Z",
          "expires_at": "2026-10-09T09:40:00Z"
        }
      }
    ]
  }
}
```

| Field                 | Meaning                                                                                                                                                                                                                                                                                                                                         |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `component`, `domain` | The component, and its domain in dotted form; `domain` is left out when the component has none                                                                                                                                                                                                                                                  |
| `gate`                | `release` or `pr`                                                                                                                                                                                                                                                                                                                               |
| `sha`, `pr_number`    | The commit the gate judged, and the pull request number; `pr_number` is left out outside a PR                                                                                                                                                                                                                                                   |
| `exercised_at`        | When the gate was consumed                                                                                                                                                                                                                                                                                                                      |
| `checks[]`            | Every gating check let through: its qualified `name`, its `enforcement` level, and the `bypass` that covered it, as the [bypasses](https://docs-lunar.earthly.dev/sql-api/views/bypasses.md) view records it. `policy` is the bypass's selector, left out when it covers every check; `expires_at` is left out for a bypass bound to its commit |

### `script_run.error`

The "a script is broken" signal: a collector, policy or cataloger run that failed. It is sent once per run, whether the Hub scheduled the run (a code, cron or cataloger run, or a policy evaluation) or the CI tracer ran the collector in a CI job:

* **The script exited non-zero**: `status` is `finished` and `exit_code` says how.
* **The Hub could not complete the run**: `status` is `internal-error` and `failure_kind` says why: `image_unavailable` or `image_pull_failed` (the image could not be pulled), `cancelled_with_batch` (another script sharing the pod could not start), `operator_cancelled` or `operator_gave_up`, `never_reported` (the run never reported a result and a stuck-run sweep ended it), or `job_failed` (the Hub job driving the run failed).

A run that fails and is then ended by the Hub, such as a non-zero exit followed by the job giving up, is still one event. A retry the Hub schedules as a new run sends a new event. An attempt the Hub retries under the same run, because the script's container was killed (exit 137, typically out of memory) or its outcome went unobserved, is not reported: either the next attempt ends the run, or the Hub gives up on the retries and ends it with `operator_gave_up`, and that is the one event. A collector the CI tracer runs is run once, so for it any non-zero exit is final.

```json
{
  "id": "f0a2...",
  "event": "script_run.error",
  "version": 1,
  "created_at": "2026-10-13T14:21:09Z",
  "data": {
    "script": "sbom",
    "type": "collector",
    "run_id": "bd80...",
    "component": "github.com/acme/api",
    "domain": "payments",
    "sha": "9c1e...",
    "pr_number": 412,
    "source": "ci",
    "ci": {
      "workflow_name": "build",
      "workflow_run_id": "38427098267",
      "workflow_run_attempt": "1"
    },
    "status": "finished",
    "exit_code": 2,
    "reason": "The script exited with code 2.",
    "started_at": "2026-10-13T14:21:02Z",
    "finished_at": "2026-10-13T14:21:08Z",
    "manifest_version": "caf908a8...",
    "run_url": "https://grafana.example.com/d/fen4vhrvim4u8b/snippet-run?var-id=bd80..."
  }
}
```

| Field                       | Meaning                                                                                                                                                                                                                                              |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `script`, `type`            | The script's name in `lunar-config.yml`, and `collector`, `policy` or `cataloger`                                                                                                                                                                    |
| `run_id`                    | The run. One event per run                                                                                                                                                                                                                           |
| `component`, `domain`       | The component the script ran for, and its domain in dotted form; `component` is left out for a global cataloger run, `domain` when the component has none                                                                                            |
| `sha`, `pr_number`          | The commit the run was for, and the pull request number; `pr_number` is left out outside a PR                                                                                                                                                        |
| `source`                    | `ci` for a collector the CI tracer ran in a CI job, `hub` for a run the Hub scheduled: on its own (a code, cron or cataloger run, a policy evaluation), or for a CI run it saw end (an after-CI-pipeline collector or policy, which runs on the Hub) |
| `ci`                        | The `workflow_name`, `workflow_run_id` and `workflow_run_attempt` of the CI run the script ran in or was run for. Always present for `source: ci`; present with `source: hub` for the scripts the Hub runs after a CI pipeline, left out otherwise   |
| `status`                    | `finished` when the script ran and exited non-zero, `internal-error` when the Hub ended the run                                                                                                                                                      |
| `exit_code`                 | The script's exit code, when it ran                                                                                                                                                                                                                  |
| `failure_kind`              | Why the Hub ended the run, one of the kinds above; left out when the script ran                                                                                                                                                                      |
| `reason`                    | The failure in words: what the Hub recorded when it ended the run, else the exit code                                                                                                                                                                |
| `started_at`, `finished_at` | When the run started and ended; each left out when unknown                                                                                                                                                                                           |
| `manifest_version`          | The configuration commit the script came from                                                                                                                                                                                                        |
| `run_url`                   | The run's page in Grafana, where its logs are; left out when the Hub has no Grafana URL configured                                                                                                                                                   |

A subscription for every failure of one kind of script, or of the scripts a team owns, is a filter away:

```bash
lunar webhooks subscribe https://hooks.example.com/lunar --event script_run.error \
    --filter '$.data.type == "policy" || $.data.domain starts with "payments"'
```

### Recovery mode transitions

`recovery_mode.entered` and `recovery_mode.exited` report committed changes to the global [recovery switch](https://docs-lunar.earthly.dev/install/lunar-hub/self-hosted/day-2-operations.md#recovery-mode-during-an-incident). Extensions, repeated exits, and replay do not create additional transition events. Recovery activation remains default-off. These experimental events are omitted from `lunar webhooks events`; subscribe using their explicit names. Existing subscriptions and exit notifications remain available after activation is disabled.

```bash
lunar webhooks subscribe https://hooks.example.com/lunar \
    --event recovery_mode.entered --event recovery_mode.exited \
    --description "Recovery notifications"
```

An entry payload includes:

```json
{
  "hub_id": "tenant-example",
  "transition_id": 42,
  "window_id": "05f0812c-d5c5-4c89-956c-993ea91b5493",
  "active": true,
  "source": "manual",
  "actor": "operator",
  "reason": "Investigating an API incident",
  "since": "2026-10-22T18:00:00Z",
  "until": "2026-10-22T19:00:00Z",
  "effective_at": "2026-10-22T18:00:00Z",
  "recorded_at": "2026-10-22T18:00:00Z",
  "retain": false,
  "enforcement_mode": "hard"
}
```

| Field                              | Meaning                                                                                                                        |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `hub_id`                           | Licence tenant identity, shared across the Hub's replicas                                                                      |
| `transition_id`, `window_id`       | Increasing ledger identifier for this transition, and the UUID shared by a window's entry and exit                             |
| `active`                           | `true` on entry, `false` on exit                                                                                               |
| `source`                           | `manual`, `automatic_budget`, `explicit_off`, or `expiry`; `unknown` can appear from an older replica during a rolling upgrade |
| `actor`, `reason`                  | The actor and reason recorded in the recovery audit ledger                                                                     |
| `since`, `until`                   | Window start and scheduled deadline; `until` is absent for an indefinite window                                                |
| `effective_at`, `recorded_at`      | When the transition took effect and when it was recorded. For expiry, the effective time is the scheduled deadline             |
| `retain`, `restore_mode`, `replay` | Retention setting and, on exit, requested restoration/replay behavior; these fields do not confirm those operations completed  |
| `enforcement_mode`                 | The window's enforcement choice: `hard`, `soft`, or `both`; it does not confirm that gate changes or check writes completed    |

The envelope's `id` and `created_at` remain stable across fan-out retries; `created_at` is the effective transition time. The producer normally runs every five seconds. Expiry notification can wait for the expiry worker, normally one minute, or a subsequent switch change; outages can delay either further. Notifications continue while recovery is active, using the usual signature, filter, pause, and receiver-retry rules.

Delivery is at least once and arrival order can change under retries. Deduplicate by envelope `id` and compare `transition_id` within a Hub to order changes. Subscribers present at fan-out receive the event; subscribing later does not replay previously handled transitions. Upgrading does not send pre-existing audit history.

## Subscribing

```bash
lunar webhooks subscribe https://hooks.example.com/lunar \
    --event bypass.exercised --description "Change-management ledger"
```

The Hub generates a secret for the subscription and prints it once. Store it in the receiver to verify signatures; it cannot be shown again. The Hub then sends the endpoint a `ping`, and `lunar webhooks delivery <id>` shows how it went.

### Changing a subscription

`lunar webhooks update` changes a subscription's events, filter and description in place. It keeps its ID, URL and secret, so the receiver has nothing new to install and its deliveries stay in one history. Whatever is not given is left as it is:

```bash
lunar webhooks update <subscription-id> --add-event script_run.error
lunar webhooks update <subscription-id> --remove-event bypass.exercised
lunar webhooks update <subscription-id> --event bypass.exercised --event script_run.error   # replace the set
lunar webhooks update <subscription-id> --filter '$.data.domain starts with "payments"'
lunar webhooks update <subscription-id> --filter ''                                        # drop the filter
lunar webhooks update <subscription-id> --description 'SOC2 evidence channel'
```

A subscription always keeps `ping`, and must keep at least one event. The change applies to the next event: a delivery already queued for an event removed here still goes out, as it was accepted when the event happened. An update does not resume a paused subscription.

To change the URL, subscribe the new endpoint and unsubscribe the old one: it is a different receiver, with its own secret.

## What the receiver gets

```http
POST /lunar HTTP/1.1
Content-Type: application/json
User-Agent: Lunar-Hookshot/<version>
X-Lunar-Event: ping
X-Lunar-Delivery: 0d4c5d8e-...
X-Lunar-Hook-ID: 7b1f9c2e-...
X-Lunar-Signature-256: sha256=<hex HMAC-SHA256 of the body, keyed with the secret>

{"id": "c3e1...", "data": {"events": ["bypass.exercised", "ping"], "subscription_id": "7b1f9c2e-..."}, "event": "ping", "version": 1, "created_at": "2026-10-08T10:12:03Z"}
```

| Header                  | Meaning                                                                                                                                    |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `X-Lunar-Event`         | The event name                                                                                                                             |
| `X-Lunar-Delivery`      | This delivery. The same across its retries and redeliveries, so a receiver can drop a repeat                                               |
| `X-Lunar-Hook-ID`       | The subscription                                                                                                                           |
| `X-Lunar-Signature-256` | `sha256=` and the hex HMAC-SHA256 of the raw body, keyed with the subscription's secret. The same scheme as GitHub's `X-Hub-Signature-256` |

The body is the envelope: `id` (the event, the same for every subscription it went to), `event`, `version`, `created_at` and the event's `data`.

### Verifying the signature

Compute the HMAC over the raw request body, before parsing it, and compare in constant time:

```python
import hashlib, hmac

def verified(secret: str, body: bytes, header: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header)
```

## Delivery and retries

* A `2xx` answer within 10 seconds (`HUB_EVENT_DELIVERY_TIMEOUT`) counts as delivered. Redirects are not followed.
* Anything else is retried after 5s, 15s, 30s and 1m, then with a widening backoff, for 8 attempts in all (about an hour and a quarter). After the last, the delivery is marked `failed`.
* After 50 consecutive failed deliveries (`HUB_EVENT_PAUSE_AFTER_FAILURES`), the subscription is paused. Events that arrive while it is paused are recorded but not sent. `lunar webhooks ping` still reaches a paused endpoint, so you can check it before running `lunar webhooks resume`.
* `lunar webhooks deliveries` lists deliveries newest first, narrowed with `--subscription`, `--event` and `--status`: `--status failed` finds the ones to redeliver.
* `lunar webhooks redeliver <delivery-id>` sends a delivery again with the same payload. Its attempts continue the delivery's trail. It refuses a delivery that still has an attempt queued, waiting on a retry or running, since that attempt sends it anyway.

Deliveries run on their own queue and worker pool, and emitting an event never waits for an endpoint, so a slow or dead receiver cannot delay collectors or policies.

## Filters

`--filter` narrows a subscription to the events a [SQL/JSON path](https://www.postgresql.org/docs/current/functions-json.html#FUNCTIONS-SQLJSON-PATH) predicate accepts. It is evaluated against the envelope, so both `$.event` and the `$.data` fields listed by `lunar webhooks events` are addressable:

```bash
lunar webhooks subscribe https://hooks.example.com/lunar --event bypass.exercised \
    --filter '$.data.gate == "release" && $.data.component starts with "github.com/acme/"'
```

| To Match                | Write                                                          |
| ----------------------- | -------------------------------------------------------------- |
| A value                 | `$.data.gate == "release"`, `!=`, `<`, `>`, `<=`, `>=`         |
| A prefix                | `$.data.component starts with "github.com/acme/"`              |
| A pattern               | `$.data.component like_regex "^github\\.com/acme/(api\|web)$"` |
| Any element of an array | `exists($.data.checks[*] ? (@.bypass.self_bypass == true))`    |
| Combinations            | `&&`, `\|\|`, `!` and parentheses                              |

A filter that does not parse is rejected when you subscribe. A filter that cannot decide on an event, because a key is missing or the result is not a boolean, rejects it: a filter never lets through what it was meant to keep out. An event a filter rejects is recorded as `filtered`, so "why didn't my endpoint get it?" has an answer.

Try a filter against recent deliveries before you subscribe with it. Nothing is sent:

```bash
lunar webhooks filter-test '$.data.gate == "release"' --event bypass.exercised
```

## Dashboards

**Platform → Webhooks** in the top menu opens **Subscriptions**: each endpoint by scheme and host, its events and state, its delivered, failed and filtered counts over the last 7 days, and when it last got a delivery.

Clicking an endpoint opens its **Subscription details**: the same summary, the filter its events must match when it has one, and every delivery made for it, newest first and filterable by event and status, each with its payload and last error. A `filtered` or `paused` delivery is an event the endpoint deliberately did not get. Clicking a time opens the delivery.

**Delivery details** shows one delivery: its row from Subscription details, then every attempt, newest first, with its time, duration, HTTP status, payload and error. What the receiver answered is not shown; `lunar webhooks delivery <id>` shows it.

## Payload compatibility

Payloads are an API. The envelope carries `"version": 1`, and payloads change only by gaining optional fields: an existing field is never renamed, removed or retyped. A breaking change would be a new event name, sent alongside the old one for a deprecation period. Ignore fields you do not know.

## Security

* For many receivers the URL is itself a credential (Slack, Teams and Discord incoming webhooks, Zapier, `https://user:pass@host`). It is stored encrypted, and only its scheme and host appear in the SQL API and Grafana. `lunar webhooks ls` shows the full URL only to `hub.operate` holders, the people who can create a subscription.
* What a receiver answers is kept, up to 4 KiB per attempt, and shown only by `lunar webhooks delivery <id>`, never in the SQL API or Grafana.
* Verify `X-Lunar-Signature-256` in the receiver: anyone who can reach the endpoint can POST to it.

## Configuration

| Variable                               | Default | Governs                                                                                                                                                 |
| -------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `HUB_EVENT_DELIVERY_TIMEOUT`           | `10s`   | How long one attempt may take, from connecting to the end of the answer                                                                                 |
| `HUB_EVENT_PAUSE_AFTER_FAILURES`       | `50`    | Consecutive failed deliveries after which a subscription is paused                                                                                      |
| `HUB_EVENT_DELIVERY_RETENTION`         | `720h`  | How long deliveries and their attempts are kept. See [Data Retention](https://docs-lunar.earthly.dev/configuration/hub-configuration/data-retention.md) |
| `HUB_EVENT_ALLOW_PRIVATE_DESTINATIONS` | `false` | Allow deliveries to private and internal network addresses, which are refused by default                                                                |

The SQL API exposes subscriptions, deliveries and attempts as the [`event_subscriptions`, `event_deliveries` and `event_delivery_attempts`](https://docs-lunar.earthly.dev/sql-api/views/webhooks.md) views.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs-lunar.earthly.dev/configuration/hub-configuration/webhooks.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
