> For the complete documentation index, see [llms.txt](https://docs-lunar.earthly.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs-lunar.earthly.dev/install/lunar-hub/self-hosted/day-2-operations.md). # Day-2 Operations This page covers what you need to run Lunar Hub after the initial install: upgrades, secret rotation, observability, and uninstall. For the per-version upgrade notes (what values changed, what to migrate), see the [chart README's Upgrading section](https://github.com/earthly/charts/blob/main/README.md#upgrading). ## Upgrading ```bash helm repo update helm upgrade lunar earthly/lunar \ --namespace lunar \ --timeout 30m \ -f values.yaml ``` Migrations run once per release as a **pre-rollout Job** (`lunar-hub-migrate`), installed via a Helm pre-install/pre-upgrade hook that completes — and gates the rollout — before any new Hub pod is created. Serving pods then **assert the schema is current at boot and refuse to start if it's behind**, which also keeps scaled-up and restarted pods safe (they don't re-run the Job). Migrations are forward-only and an interrupted one is safe to retry. {% hint style="warning" %} **Allow time for concurrent index builds.** Helm's [`--timeout` defaults to five minutes](https://helm.sh/docs/helm/helm_upgrade/), including its wait for the migration hook. The example above allows 30 minutes; larger databases or busy transactions can require more. PostgreSQL's [concurrent index builds](https://www.postgresql.org/docs/17/sql-createindex.html#SQL-CREATEINDEX-CONCURRENTLY) wait for existing transactions as well as scanning the table. Keep `hub.migrateJobActiveDeadlineSeconds` above the Helm timeout; its default is `3600` seconds. Increasing only `--timeout` does not extend this Kubernetes deadline. If you use Terraform's `helm_release`, set its `timeout` to the equivalent number of seconds. {% endhint %} If Helm times out, inspect the migration Job before retrying: ```bash kubectl -n lunar get job lunar-hub-migrate kubectl -n lunar logs job/lunar-hub-migrate --all-containers=true ``` The Job can continue after Helm stops waiting. Wait for a healthy migration to finish before rerunning the upgrade. A new upgrade deletes the previous hook Job through `before-hook-creation`; interrupting an index build makes the next attempt drop the unfinished index and rebuild it. If the Job has failed, fix the cause and increase the timeout or Job deadline as needed before retrying. **Before every upgrade:** 1. **Back up Postgres.** Migrations are forward-only. If you need to roll back, roll the database back. 2. **Preview the chart changes.** The [`helm-diff`](https://github.com/databus23/helm-diff) plugin is worth installing — `helm diff upgrade lunar earthly/lunar -f values.yaml` shows exactly what Kubernetes resources will change. 3. **Read the version-specific notes** in the [chart README](https://github.com/earthly/charts/blob/main/README.md#upgrading). Minor version bumps occasionally require values changes. 4. **Pin your new image tags** in `values.yaml`. Don't rely on the chart's default tags. {% hint style="warning" %} Migrations are forward-only — back up Postgres first (step 1). Author schema changes **expand/contract** (add first, drop in a later release) so the common upgrade stays zero-downtime: the migrate Job runs while the old-version pods keep serving, then Helm rolls the Deployment. Reserve a maintenance window only for an unavoidable breaking change, and confirm the budget with stakeholders — a heavy migration can take tens of minutes, during which the old version keeps serving. {% endhint %} With multiple replicas and a PodDisruptionBudget, Helm's rolling update replaces Hub pods a few at a time while the rest keep serving, so a routine upgrade has no API-unavailability window. ## Preflight checks With `HUB_PREFLIGHT_EXIT=true`, the Hub checks that it can reach its database, S3 buckets, git platforms, and telemetry endpoints, prints the results, and exits instead of starting. `OPERATOR_PREFLIGHT_EXIT=true` does the same for the Kubernetes permissions the snippet operator starts script pods with. Both help when you move Lunar from one cluster to another: ```bash kubectl -n lunar exec deploy/lunar-hub -- env HUB_PREFLIGHT_EXIT=true lunar-hub kubectl -n lunar exec deploy/lunar-operator -- env OPERATOR_PREFLIGHT_EXIT=true lunar-snippet-operator ``` Each prints a line per check and exits 1 if any fails. To run the checks at every startup and only log the results, set `HUB_PREFLIGHT_ENABLED=true` or `OPERATOR_PREFLIGHT_ENABLED=true` instead. ## Rotating secrets Lunar's secrets split into two categories — **user-managed** (you create and rotate them) and **chart-managed** (auto-generated on first install, preserved across upgrades via `helm.sh/resource-policy: keep`). See [Required secrets](https://github.com/earthly/charts/blob/main/README.md#required-secrets) in the chart README for the full list and naming rules. ### Hub auth token (`-auth-token`) Used by the CLI and CI tracers to authenticate to the Hub. When the Hub boots with a different token from the one it has stored, it keeps the old one working for a grace period so its holders have time to move over. The grace period is one week by default, set by `HUB_AUTH_TOKEN_PREVIOUS_VALID_FOR` in `hub.extraEnv`. To rotate, delete the secret and re-run `helm upgrade` so the chart generates a fresh token, then restart the Hub and the Operator: ```bash kubectl -n lunar delete secret lunar-auth-token helm upgrade lunar earthly/lunar -n lunar -f values.yaml kubectl -n lunar rollout restart deployment/lunar-hub kubectl -n lunar rollout restart deployment/lunar-operator ``` Both read the token from the same secret (the Operator uses it as `OPERATOR_HUB_TOKEN` to call the Hub), so restart both. The Hub logs `HUB_AUTH_TOKEN changed; the previous token is still accepted until it expires` with the exact expiry. Retrieve the new token with `kubectl -n lunar get secret lunar-auth-token -o jsonpath='{.data.token}' | base64 -d`, then update every CI tracer's `LUNAR_HUB_TOKEN` and every developer's CLI config before that time. {% hint style="warning" %} **Suspect a leak? Set the grace period to zero.** Add `HUB_AUTH_TOKEN_PREVIOUS_VALID_FOR=0` to `hub.extraEnv` in the same `helm upgrade`, and the old token stops working as soon as the first new Hub pod starts. Builds still using it fail and need to be retried. {% endhint %} To end a grace period early, expire the old token in the database: ```sql UPDATE hub.auth_tokens SET expires_at = now() AT TIME ZONE 'UTC' WHERE name LIKE 'default-retired-%'; ``` Rotation only moves forward. A token the Hub has retired is never reinstated, even if you restore the old secret, so roll back by generating a new token instead. To pin to an externally-managed token instead, set `hub.auth.secretName` in values to a secret you control — the chart will consume it instead of generating its own. ### Git platform credentials Both are read once at boot, so a rolling restart is always required after updating the secret. {% tabs %} {% tab title="GitHub" %} **GitHub App private key (`lunar-github-app`).** GitHub allows multiple active keys per App, so this rotates with zero downtime: 1. **Generate a new key** in GitHub → **Apps → your App → Private keys → Generate a private key**. Download the new PEM. 2. **Update the Kubernetes secret** in place (the Hub expects a base64-encoded PEM inside the secret value): ```bash kubectl -n lunar create secret generic lunar-github-app \ --from-literal=private-key="$(base64 < path/to/new-key.pem | tr -d '\n')" \ --dry-run=client -o yaml | kubectl apply -f - ``` 3. **Roll the Hub** — `kubectl -n lunar rollout restart deployment/lunar-hub`. 4. **Verify** webhooks still deliver (GitHub → any repo → **Settings → Webhooks → Recent Deliveries**). 5. **Delete the old key** in GitHub. {% endtab %} {% tab title="GitLab" %} **Group access token (`lunar-gitlab-token`).** Revoking the old token takes effect immediately, so revoke it last: 1. **Create the replacement token** in GitLab (group → **Settings → Access tokens**), with the same scope and role as the original. 2. **Update the secret in place:** ```bash kubectl -n lunar create secret generic lunar-gitlab-token \ --from-literal=token='' \ --dry-run=client -o yaml | kubectl apply -f - ``` 3. **Roll the Hub** — `kubectl -n lunar rollout restart deployment/lunar-hub`. 4. **Revoke the old token** in GitLab. {% hint style="warning" %} GitLab access tokens expire on a fixed date and nothing renews them. When one lapses, Lunar silently stops posting results and reacting to webhooks for that group. Put the expiry date in a calendar and rotate ahead of it. {% endhint %} {% endtab %} {% endtabs %} ### Webhook secrets The chart generates one of these per platform. Lunar Hub moves each hook to the new secret on its own, so a rotation only needs the old value kept accepted until that has happened. {% hint style="warning" %} **Suspect a leak? Don't keep the old value.** Replace the secret without listing the old one as previous. Each hook is still repaired, but its deliveries are rejected until then, which can take hours on a large fleet. {% endhint %} {% stepper %} {% step %} ### Replace the secret, keeping the old value accepted Copy the current value into a Secret of its own and list it as the previous one, then let the chart generate a new secret and restart. The commands are for GitHub; for GitLab, use `lunar-gitlab-webhook` and `HUB_GITLAB_WEBHOOK_SECRET_PREVIOUS`. ```bash kubectl -n lunar create secret generic lunar-github-webhook-previous \ --from-literal=webhook-secret="$(kubectl -n lunar get secret lunar-github-webhook -o jsonpath='{.data.webhook-secret}' | base64 -d)" ``` {% code title="values.yaml" %} ```yaml hub: extraEnv: - name: HUB_GITHUB_WEBHOOK_SECRET_PREVIOUS valueFrom: secretKeyRef: name: lunar-github-webhook-previous key: webhook-secret ``` {% endcode %} ```bash kubectl -n lunar delete secret lunar-github-webhook helm upgrade lunar earthly/lunar -n lunar -f values.yaml kubectl -n lunar rollout restart deployment/lunar-hub ``` To rotate a per-App `webhook_secret` in `HUB_GITHUB_APPS` or a per-group one in `HUB_GITLAB_TOKENS`, set the new value on that entry and move the old one to its `previous_webhook_secrets` instead. {% endstep %} {% step %} ### Wait for the hooks to move Nothing to do. Each hook is re-stamped with the new secret on its next delivery, and nothing is lost meanwhile. A repository or project with no activity keeps the old secret until it next delivers, so keep the old value accepted for a few weeks. {% endstep %} {% step %} ### Retire the old value Remove the `HUB_*_WEBHOOK_SECRET_PREVIOUS` entry (or the old `previous_webhook_secrets` value), `helm upgrade`, restart, and delete `lunar-github-webhook-previous`. A hook that still carried the old value has its next delivery rejected, and is repaired. {% endstep %} {% endstepper %} ### Database password (`lunar-db`) Standard Kubernetes secret rotation. If your Postgres supports multiple active passwords (e.g. RDS password grace), rotate without downtime: 1. Set the new password in Postgres (keeping the old one active). 2. Update the `lunar-db` secret. 3. Restart the Hub. 4. Revoke the old password. If your Postgres doesn't support multiple active passwords, expect \~1 minute of Hub unavailability during the rotation. ### Other secrets A few secrets don't get their own recipe above: | Secret | Managed by | Consumed by | | -------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------- | | `-grafana-admin` | Chart | The bundled Grafana + the Hub (only in `grafana.mode: chart`) | | `-grafana-db` — read-only `grafana_user` role | Chart | Grafana datasource — whenever dashboards deploy (bundled **or** your own) | | Your own Grafana login (`HUB_GRAFANA_TOKEN`, or `HUB_GRAFANA_USER` + `HUB_GRAFANA_PASSWORD`) | You | Hub — only with your own Grafana ([Option B/C](/install/lunar-hub/self-hosted/install-walkthrough.md#grafana)) | | Hub licence JWT (example `lunar-hub-licence`) | You | Hub | | Per-scope runtime secrets (`hub.secrets.{collector,cataloger,policy}.secretName`) | You | Hub | The rotation shape follows the same split as the sections above: * **Chart-managed** (Grafana admin): `kubectl delete secret ` → `helm upgrade` to regenerate → restart the consumer deployment. This does **not** apply to `-grafana-db` — treat it as immutable after install. * **User-managed** (hub licence JWT, runtime secrets, your own Grafana credentials): update the secret in place (`kubectl apply` or `kubectl create --dry-run=client -o yaml | kubectl apply -f -`) → restart the consumer(s). After rotating your own Grafana credentials, restart the Hub so it picks up the new value — the next `helm upgrade` re-runs the chart's dashboard-deploy Job with it. To find what consumes an arbitrary secret: ```bash kubectl -n lunar get deployments -o yaml | grep -B2 '' ``` ## Webhook health Lunar registers a webhook on every tracked repo and re-verifies it periodically. A hook that was disabled — by GitLab after the Hub was unreachable or too slow to answer for a stretch, or by a repo admin unticking **Active** on GitHub — is switched back on at the next re-sync (at worst `HUB_WEBHOOK_FRESHNESS`, default 12h, plus one sync pass). Events sent while a hook was off are not replayed. Deliveries the Hub rejects or fails to process are tracked in [`lunar_webhook_deliveries_total`](#webhook-deliveries). ## Observability ### Logs The Hub and Operator log in structured JSON by default (`logging.format: json`). Log level defaults to `info`; raise to `debug` temporarily by editing `values.yaml` and running `helm upgrade`. The Hub and Operator don't ship logs anywhere themselves — they stream to stdout. To get them into your log aggregator, point a cluster-level log shipper (Fluent Bit, Vector, Datadog Agent, etc.) at the `lunar` namespace. ### Telemetry to Earthly Hub tenant identity and telemetry destinations come from the signed licence JWT, not from `HUB_TENANT_ID`, `HUB_ELASTIC_*`, or `HUB_OTEL_*` environment variables. The chart handles the mount and `HUB_LICENCE_FILE` wiring for you. Keep (or override) the licence secret reference in values: ```yaml hub: licence: secretName: lunar-hub-licence secretKey: hub-licence.jwt # optional; defaults to /var/run/secrets/lunar/hub-licence.jwt # filePath: /var/run/secrets/lunar/hub-licence.jwt ``` Rotation flow: * Request a new JWT from Earthly. * Update the Kubernetes secret value (`hub-licence.jwt`) in place. * The Hub re-validates every 5 minutes and also enforces the exact `exp` boundary. ### Metrics The Hub exports request-duration histograms (HTTP and gRPC) via OTLP when the active licence includes `telemetry.otel.endpoint` and `telemetry.otel.token`. If `telemetry.otel` is omitted, the Hub falls back to writing metrics to a local temp file. The same instrumentation is always available to scrape at `GET /metrics` on the Hub's health port (`8002`), independent of the OTLP destination. The chart doesn't put that port on the Hub Service or ship a monitor object, so reaching it means pointing a `PodMonitor` at the container port — see [Ports and protocols](/install/lunar-hub/self-hosted/overview.md#ports-and-protocols). Secure (TLS) OTLP is not implemented yet, so OTLP export runs insecure today. {% hint style="info" %} The Hub does not export distributed traces today — only metrics. Trace support is on the roadmap. {% endhint %} ### Webhook deliveries Inbound webhook deliveries and what happened to them. For GitLab, this is where failed deliveries show up (see [Webhook health](#webhook-health)). | Metric | Type | Labels | | -------------------------------- | ------- | ----------------------------- | | `lunar_webhook_deliveries_total` | counter | `provider`, `event`, `result` | * `result` — `accepted`, `duplicate` (a redelivery of one already received), or why a delivery wasn't processed: `unhandled_type`, `bad_request`, `rejected_auth` (webhook secret mismatch), `processing_failed`, `persist_failed` (received but couldn't be stored for processing). Useful alert: ```promql # Deliveries are being rejected, or failing to be stored or processed. increase(lunar_webhook_deliveries_total{result=~"rejected_auth|processing_failed|persist_failed"}[15m]) > 0 or increase(lunar_webhook_processed_total{status=~"failed|cancelled"}[15m]) > 0 ``` ### Reporting to the git platform ("the last mile") These answer the question a queue metric can't: *did the verdict actually reach GitHub or GitLab, and in what state?* | Metric | Type | Labels | | ---------------------------------------- | --------- | -------------------------------------------------------- | | `lunar_check_posts_total` | counter | `git_platform`, `surface`, `action`, `result`, `outcome` | | `lunar_check_time_to_conclusion_seconds` | histogram | `git_platform`, `result` | `lunar_check_posts_total` increments once per reporting artifact the Hub writes — and once per artifact it deliberately *doesn't* write, which is why `action` matters: without it a suppressed comment and a lost one look identical. * `git_platform` — `github` or `gitlab`. * `surface` — `check_run`, `commit_status`, `external_status_check`, `pr_comment`, `badge`. * `action` — `created`, `updated`, or `posted` (a write the platform's API doesn't split into create vs update); `unchanged` (the artifact already said exactly this), `suppressed` (`customization.pr_comments.mode: only_failures` kept quiet), `skipped_stale_head` (the branch head moved on, and the new head is reported on its own). * `result` — the state that was reported, normalized across git platforms: `in_progress`, `success`, `failure`. Surfaces that carry no state (comments, badges) report `none`. * `outcome` — `ok`, or `error` when the git platform rejected the write. `lunar_check_time_to_conclusion_seconds` records, at the write that settles a check, how long it took from the policy post being dispatched. It covers the reporting leg only, not the whole journey from the pull request being opened. Two things are counted that you might not expect, and one that isn't: * **Break-glass answers are counted.** Granting or revoking a [`/lunar bypass`](/docs/pr-comments.md) writes to the merge gate, so it appears as an `external_status_check` post — a grant as `success`, a revoke as `in_progress`. A revoke is meant to be followed by a real evaluation, so if one never concludes, the first alert below is telling you the truth: that gate is stuck. * **Deliberate non-writes are counted**, with `action` saying which (see the glossary above). `outcome` stays `ok` for these — nothing failed. * **The GitLab pending-check heartbeat is not counted.** A blocked merge request's gate is re-asserted every 90 seconds to beat GitLab's two-minute auto-fail. Counting it would add 40 `in_progress` samples an hour per stuck merge request and turn this counter from "reports made" into "API calls made". So `lunar_check_posts_total` will read lower than GitLab's own API write count, on GitLab only — GitHub has no equivalent healer. Useful alerts: ```promql # Checks are being opened and nothing is concluding them — required checks # would sit pending indefinitely, with no failed request and no queue backlog. increase(lunar_check_posts_total{result="in_progress"}[1h]) > 0 and increase(lunar_check_posts_total{result=~"success|failure"}[1h]) == 0 # The git platform rejected a write: everything worked inside Lunar and the # result never landed. increase(lunar_check_posts_total{outcome="error"}[15m]) > 0 # Checks are settling far slower than usual. histogram_quantile(0.95, sum by (le, git_platform) (rate(lunar_check_time_to_conclusion_seconds_bucket[30m]))) > 900 ``` Two more are available without these metrics and are worth watching alongside: `max(river_jobs{kind="policy_post",state=~"retryable|discarded"}) > 0` (posts that gave up after their retries) and `increase(lunar_github_requests_total{operation="check_runs",outcome="error"}[15m]) > 0` (noisier — the same `operation` value covers the read the Hub makes before every post). ### Snippet images A snippet image that can't be pulled fails every run that uses it, and the only other place that shows up is the report on each pull request. Two counters catch it earlier: the Hub checks each image when it pulls a new configuration, and the snippet operator counts every pull its run pods make. | Metric | Exported by | Type | Labels | | --------------------------------- | --------------------------------------- | ------- | ---------------------------------- | | `lunar_image_checks_total` | Hub (`8002`) | counter | `result` | | `lunar_snippet_image_pulls_total` | Snippet operator (`/metrics` on `8080`) | counter | `result`, `reason`, `snippet_type` | * `lunar_image_checks_total` — one sample per distinct image each time the Hub pulls a configuration, unless `HUB_IMAGE_CHECK` is `off`. `result` is `ok`, `not_found` (the registry has no such tag or repository), `invalid` (the reference doesn't parse), or `unverified` (no definitive answer: missing credentials, a denied request, an unreachable registry). It counts what the registry said, so an image found within `HUB_IMAGE_CHECK_CACHE_TTL` counts as `ok` again without a lookup, and an image that only warned, because the running configuration already uses it, still counts as `not_found`. * `lunar_snippet_image_pulls_total` — `result` is `success` or `failure`. `reason` is the kubelet's reason on a failure (`ErrImagePull`, `ImagePullBackOff`, `InvalidImageName`) and empty on a success. `snippet_type` is `collector`, `policy`, `cataloger`, or `other`. Each run pod is counted once, when the operator stops waiting on it: one sample per container, init containers included. A success means the container started, so an image the node already had cached counts too. No label names the image or registry, so the series count stays fixed. Each operator replica counts the pods it ran, so `sum` across replicas. Useful alerts: ```promql # Run pods are failing to pull an image. A count, not a ratio: one bad tag # fails one container in a pod whose other containers mostly start, so even # a broken image leaves the failure ratio low. Raise the threshold if # transient registry throttling is noisy. sum(increase(lunar_snippet_image_pulls_total{result="failure"}[15m])) > 0 # A configuration names an image its registry doesn't have. increase(lunar_image_checks_total{result=~"not_found|invalid"}[1h]) > 0 ``` ### Aggregating metrics across replicas With multiple replicas, be deliberate about how dashboards and alerts aggregate the Hub's job-queue metrics. Queue-depth **gauges** (e.g. `river_jobs`) read from the queue tables in Postgres, which are **shared** by every replica, so each replica reports the same global count — aggregate with **`max`** (or `avg`), never `sum`, which multiplies the backlog by the replica count. Per-replica **counters** are the opposite: `river_insert_count_total` / `river_work_count_total` and `lunar_check_posts_total` count work done by *each* replica, so `sum(rate(...))` is the correct cluster-wide throughput. ### Diagnostics bundle The Hub can produce a diagnostics bundle (Postgres queue state, recent error logs, slow-query stats) for support requests. Generate one with: ```bash lunar hub get-logs ``` This collects `kubectl` logs from the Hub pods plus Postgres telemetry from the Hub, then writes a `lunar-hub-logs-.tar.gz` in the current directory and prints its path. Run it from a machine with `kubectl` access to the Hub's cluster and your Lunar CLI configured to reach the Hub. Common flags: * `-n`, `--namespace` — Kubernetes namespace to read Hub pods from (defaults to the current kubectl context's namespace). * `-o`, `--output` — path to write the bundle to (defaults to `lunar-hub-logs-.tar.gz` in the current directory). * `--tail` — log lines to capture per container (default: `1000`; `-1` captures the whole log). The bundle gathers extra Postgres telemetry when the `pg_stat_statements` extension is enabled — without it the bundle still works, just with less query-performance data. To enable the extension: * **Amazon RDS / Aurora:** add `pg_stat_statements` to `shared_preload_libraries` in the DB parameter group, then reboot the instance. * **Self-managed:** add `pg_stat_statements` to `shared_preload_libraries` in `postgresql.conf` and restart, then `CREATE EXTENSION pg_stat_statements;` as a superuser. The Hub does not require this extension — it's only used by the diagnostics path. ## Limit Grafana query time [`uiPanels`](/configuration/lunar-config/uipanels.md) run configuration-authored SQL whenever someone views the panel. Their row limit controls browser output but does not limit database work. Set `HUB_GRAFANA_STATEMENT_TIMEOUT` through `hub.extraEnv` to limit query duration, for example to `10s`. The setting applies to the Grafana database role, so it covers bundled dashboard queries as well as UI panels. An unset variable means no timeout. If a UI panel query exceeds the limit, Postgres cancels the whole panel query and Grafana displays the error. The database migration applies this role setting, and the chart's migrate Job inherits `hub.extraEnv`. The timeout affects database sessions opened after the migration completes. Grafana can retain pooled connections for up to four hours; re-run dashboard provisioning or restart Grafana to apply the new timeout immediately. ## Scaling The Hub is stateless — all durable state lives in Postgres and S3 — so it scales **horizontally**: raise `hub.replicaCount` and the replicas coordinate through Postgres (leader election, advisory locks) with no peer-to-peer wiring. Size the connection pool against your Postgres `max_connections` first — see [Replicas and the Postgres connection budget](/install/lunar-hub/self-hosted/install-walkthrough.md#replicas-and-the-postgres-connection-budget). Add a PodDisruptionBudget (`hub.podDisruptionBudget.enabled: true`) so node drains and rollouts keep a quorum. Vertical scaling still helps a single busy replica: CPU, memory, and the Postgres pool via `hub.db.maxOpenConns` / `hub.db.maxPoolConns`. Run throughput scales independently — raise `operator.maxConcurrent` and size run pod resources accordingly. See [`operator.scriptContainerSpec*`](https://github.com/earthly/charts/blob/main/README.md#values-reference) in the chart README. ### Leader election and periodic jobs Exactly one replica is the elected **leader** at a time; it runs the periodic and maintenance jobs (cleanup, reconcile, backfill). Jobs marked `RunOnStart` fire **immediately whenever leadership is (re)acquired** — not just at process boot — so a failover re-runs them on the new leader. Their handlers must be **idempotent**; the current `RunOnStart` jobs already are. Don't add one whose effect isn't safe to repeat on an arbitrary leadership flip. ## Prioritizing core services over script pods By default the Operator creates script pods (collectors, policies, catalogers) at the cluster's default priority — the same priority as the Hub, Operator, and Grafana. When those script pods share a node with the core services, a burst of runs can drive the node into memory pressure and the kubelet may evict a *core* pod when it should be shedding an ephemeral script instead. This matters in two setups: * **Single-namespace installs** — the Operator runs scripts in the same namespace (and usually the same node group) as the Hub, so core and ephemeral workloads compete for the same memory. * **Shared scratch namespaces or node groups** — scripts land on nodes shared with other workloads, and you want Lunar's ephemeral pods to yield first under contention. `operator.scriptPodPriorityClassName` (chart `>= 2.3.0`) sets the [PriorityClass](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/) on every script pod the Operator creates. Point it at a class whose value sits *below* your core services, and the kubelet evicts scripts first under pressure — the scheduler also won't preempt a higher-priority pod to make room for a script. ### Setup 1. **Create the PriorityClass.** The chart references it by name but does not create it. ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: lunar-script value: -10 # negative → below the default (0) your core pods use globalDefault: false description: Ephemeral Lunar script pods; evicted before core services. ``` ```bash kubectl apply -f lunar-script-priorityclass.yaml ``` 2. **Point the Operator at it** in `values.yaml`: ```yaml operator: scriptPodPriorityClassName: lunar-script ``` 3. **Upgrade and restart the Operator** — only script pods created *after* the restart inherit the class: ```bash helm upgrade lunar earthly/lunar -n lunar -f values.yaml kubectl -n lunar rollout restart deployment/lunar-operator ``` Leave the Hub, Operator, and Grafana on the cluster default priority — don't set a class on them. The gap between their `0` and the script class's negative value is what orders eviction; you lower scripts rather than raise core services. In a shared scratch node group, pick a value below whatever neighbouring workloads run at so Lunar's scripts are the first thing to go. {% hint style="warning" %} PriorityClasses are **cluster-scoped**, and a pod that references one that doesn't exist is rejected at admission — so the Operator will fail to launch scripts until the class is applied. Create the class *before* (or in the same change as) the `helm upgrade`, and delete it only once no workload references it. {% endhint %} {% hint style="info" %} Priority orders *who gets evicted first* — it is not a resource cap. Keep your script pod requests/limits set (see [`operator.scriptContainerSpec*`](https://github.com/earthly/charts/blob/main/README.md#values-reference) in the chart README) so a single run can't consume a whole node before the kubelet reacts. Priority decides the order; requests and limits decide the ceiling. {% endhint %} ## Operating the GitLab merge gate If you run [the merge gate](/install/git-platforms/gitlab.md#merge-gate) on GitLab Ultimate, two of its properties are operational rather than install-time: * **It is fail-closed.** A Hub outage leaves gated merge requests blocked. That is the correct default for a compliance control, but it means a Lunar incident becomes a merge freeze. * **The immediate lever is on the GitLab side.** To unblock merges during an incident, a project Owner disables **Status checks must succeed** in the project's merge-request settings. Lunar re-enables it on the next sync, so this buys time rather than being an opt-out. It is also the only lever that works while the Hub is down; everything below is applied by the Hub. A project that should be permanently ungated belongs out of Lunar's configuration instead: once its last component leaves the catalog, the Hub deletes the `Earthly Lunar` status check on its next sweep (`HUB_GITLAB_MERGE_GATE_REAP_INTERVAL`, 1 minute), at most `HUB_GITLAB_MERGE_GATE_REAP_BATCH_SIZE` projects (1000) per sweep. There is no undo window by default; `HUB_GITLAB_MERGE_GATE_REAP_GRACE` adds one if you want the check to survive a catalog change that is reverted within it, since re-adding the component re-creates the check and marks every open merge request `passed`. The project's webhook follows on the webhook reaper's own 24-hour schedule. * **The fleet-wide switch is on the Hub.** `HUB_GITLAB_MERGE_GATE_ENABLED=false` pauses the gate everywhere: the Hub removes the `Earthly Lunar` check from every project it registered one on, at most `HUB_GITLAB_MERGE_GATE_REAP_BATCH_SIZE` projects per sweep, and provisions no new ones. A change takes effect about three sweeps after the rollout completes (the replicas of a rolling deploy must agree before the new value is applied, so a half-rolled fleet cannot tear down and re-create the same checks in turn) and then works through the projects, most recently active first, one batch per sweep: minutes for a few hundred projects, within the hour for thousands. Switching on is the slower direction, since re-creating a check also answers each of the project's open merge requests. Two transition effects to expect: switching off moves open merge requests' verdicts to the commit status, which gates nothing unless **Pipelines must succeed** is on; switching on re-creates the check and answers `passed` for every merge request open at that moment. **Status checks must succeed** is never changed by the switch. `HUB_GITLAB_BADGES_ENABLED=false` does the same for Lunar's project badges: removed everywhere, and not posted again until it is back on. Do not re-create a removed check by hand; only the Hub's own creation grandfathers the open merge requests. * **The runtime switch is in the CLI.** [`lunar hub merge-gate off`](/docs/lunar-cli.md#lunar-hub-merge-gate) switches the check and the badges off for every project or for a group, with no redeploy, for anyone with [`hub.operate`](/docs/authorization.md#hub-operate) (configuration-repository Maintainers by default). `on --group G` underneath a broader off pins G back on for as long as that off stands, so a staged rollout is one fleet-wide off and one `on` per group. Every off expires: `--for` defaults to, and is capped by, `HUB_MERGE_GATE_OFF_MAX` (default `720h`; set it on the Hub here, or ask Earthly on Lunar Dedicated). The environment flags stay hard kill switches underneath: with `HUB_GITLAB_MERGE_GATE_ENABLED=false`, `on` retires the standing off but the check returns only when the flag does (badges follow `HUB_GITLAB_BADGES_ENABLED` and return at once). Every change lands in `hub.merge_gate_changes` with who, why and until when. * **What the Hub found is on record.** Each repository row in `hub.git_repositories` carries `merge_gate_check_id`, `merge_gate_reconciled_at` and `merge_gate_reconcile_result` (`ensured`, `removed`, `unlicensed`, `unreachable`), refreshed after a switch and at least daily. `SELECT merge_gate_reconcile_result, count(*) FROM hub.git_repositories WHERE host = 'gitlab.com' GROUP BY 1` over the SQL API tells you where the fleet stands; every row turning `unreachable` inside a day means the token, not the projects. On Lunar Dedicated, Earthly holds the switches and you have the same record. Individual blocked merge requests do not need an incident response. An authorized engineer can comment [`/lunar bypass: `](/docs/pr-comments.md) on the merge request, which is recorded and applies to that commit only. ## Backup and disaster recovery Lunar's authoritative state lives in the systems you already back up: * **Postgres** — use your existing Postgres backup process. Everything the Hub cares about (components, policies, run history, queue state) is here. * **S3 buckets** — enable object versioning on both buckets and, if compliance requires, cross-region replication. Lost resource archives cause re-building the cache of catalogers, collectors, and policies; lost log archives mean lost history. Neither impedes Hub operation. * **Hub local scratch** — each Hub pod's `/var/lib/lunar` is an ephemeral `emptyDir` (re-extracted runtimes and cached bundles; the authoritative copy is in S3). It is per-pod and regenerable, so there's nothing to back up. * **Kubernetes secrets** — keep your Git platform credential (the GitHub App PEM or the GitLab group access token), DB credentials, auth token, and webhook secret in your organization's secret-management system of record. Losing them means re-provisioning. ## Uninstalling ```bash helm uninstall lunar --namespace lunar ``` Helm removes all resources the chart created: the Hub and operator deployments, services, ingress, RBAC, and service accounts. **Not removed automatically:** * The `lunar` namespace itself (and, if used, the Operator's execution namespace). * Kubernetes secrets you created manually (e.g. `lunar-db`, `lunar-github-app`, `lunar-gitlab-token`, `lunar-hub-licence`). * **Anything Lunar provisioned on your GitLab projects** — webhooks, the `Earthly Lunar` status check, and the **Status checks must succeed** setting. Uninstalling the Hub does not undo them, and with the Hub gone nothing answers the status check, so gated merge requests stay blocked. Before uninstalling, disable that setting on the affected projects. * Chart-managed secrets (`-auth-token`, `-github-webhook`, `-gitlab-webhook`, `-grafana-admin`). These carry `helm.sh/resource-policy: keep` so a re-install reuses the same values — by design. Delete them explicitly if you want fresh credentials on the next install. * Your Postgres database, S3 buckets, or their contents. * The GitHub App. Webhooks previously registered by this Hub are tagged with the Hub's instance ID (`earthly-lunar-` as a URL fragment); they remain on your repos until you remove them manually or install a fresh Hub with the same tenant ID, which will clean up its own stale webhooks on next config pull. To fully tear down: ```bash helm uninstall lunar --namespace lunar kubectl -n lunar delete secret lunar-db lunar-github-app lunar-gitlab-token \ lunar-hub-licence \ lunar-auth-token lunar-github-webhook lunar-gitlab-webhook lunar-grafana-admin \ --ignore-not-found kubectl delete namespace lunar # Externally: # - Drop the Postgres database # - Delete or empty the S3 buckets # - Uninstall the GitHub App from your organization # - Revoke the GitLab group access token, and remove the status check and # "Status checks must succeed" setting from any gated projects ``` ## Getting help * [Chart values reference](https://github.com/earthly/charts/blob/main/README.md#values-reference) * [Chart source](https://github.com/earthly/charts) * [Lunar source](https://github.com/earthly/lunar) * For enterprise onboarding or production sizing guidance, [contact the Earthly team](https://earthly.dev/earthly-lunar/demo). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs-lunar.earthly.dev/install/lunar-hub/self-hosted/day-2-operations.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.