> For the complete documentation index, see [llms.txt](https://docs-lunar.earthly.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs-lunar.earthly.dev/install/lunar-hub/self-hosted/day-2-operations.md). # Day-2 Operations This page covers what you need to run Lunar Hub after the initial install: upgrades, secret rotation, observability, and uninstall. For the per-version upgrade notes (what values changed, what to migrate), see the [chart README's Upgrading section](https://github.com/earthly/charts/blob/main/README.md#upgrading). ## Upgrading ```bash helm repo update helm upgrade lunar earthly/lunar \ --namespace lunar \ -f values.yaml ``` Migrations run once per release as a **pre-rollout Job** (`lunar-hub-migrate`), installed via a Helm pre-install/pre-upgrade hook that completes — and gates the rollout — before any new Hub pod is created. Serving pods then **assert the schema is current at boot and refuse to start if it's behind**, which also keeps scaled-up and restarted pods safe (they don't re-run the Job). Migrations are forward-only and an interrupted one is safe to retry. **Before every upgrade:** 1. **Back up Postgres.** Migrations are forward-only. If you need to roll back, roll the database back. 2. **Preview the chart changes.** The [`helm-diff`](https://github.com/databus23/helm-diff) plugin is worth installing — `helm diff upgrade lunar earthly/lunar -f values.yaml` shows exactly what Kubernetes resources will change. 3. **Read the version-specific notes** in the [chart README](https://github.com/earthly/charts/blob/main/README.md#upgrading). Minor version bumps occasionally require values changes. 4. **Pin your new image tags** in `values.yaml`. Don't rely on the chart's default tags. {% hint style="warning" %} Migrations are forward-only — back up Postgres first (step 1). Author schema changes **expand/contract** (add first, drop in a later release) so the common upgrade stays zero-downtime: the migrate Job runs while the old-version pods keep serving, then Helm rolls the Deployment. Reserve a maintenance window only for an unavoidable breaking change, and confirm the budget with stakeholders — a heavy migration can take tens of minutes, during which the old version keeps serving. {% endhint %} With multiple replicas and a PodDisruptionBudget, Helm's rolling update replaces Hub pods a few at a time while the rest keep serving, so a routine upgrade has no API-unavailability window. ## Rotating secrets Lunar's secrets split into two categories — **user-managed** (you create and rotate them) and **chart-managed** (auto-generated on first install, preserved across upgrades via `helm.sh/resource-policy: keep`). See [Required secrets](https://github.com/earthly/charts/blob/main/README.md#required-secrets) in the chart README for the full list and naming rules. ### Hub auth token (`-auth-token`) Used by the CLI and CI tracers to authenticate to the Hub. The Hub accepts a single token today — rotation requires a coordinated cutover. To force a regeneration, delete the secret and re-run `helm upgrade` (the chart will generate a fresh token): ```bash kubectl -n lunar delete secret lunar-auth-token helm upgrade lunar earthly/lunar -n lunar -f values.yaml kubectl -n lunar rollout restart deployment/lunar-hub kubectl -n lunar rollout restart deployment/lunar-operator ``` Both the Hub and the Operator read this token from the same secret (the Operator uses it as `OPERATOR_HUB_TOKEN` to call the Hub), so restart both. Retrieve the new token with `kubectl -n lunar get secret lunar-auth-token -o jsonpath='{.data.token}' | base64 -d`, then update every CI tracer's `LUNAR_HUB_TOKEN` and every developer's CLI config. Active builds authenticated with the old token will fail mid-run and need to be retried. Dual-token support is on the roadmap. To pin to an externally-managed token instead, set `hub.auth.secretName` in values to a secret you control — the chart will consume it instead of generating its own. ### Git platform credentials Both are read once at boot, so a rolling restart is always required after updating the secret. {% tabs %} {% tab title="GitHub" %} **GitHub App private key (`lunar-github-app`).** GitHub allows multiple active keys per App, so this rotates with zero downtime: 1. **Generate a new key** in GitHub → **Apps → your App → Private keys → Generate a private key**. Download the new PEM. 2. **Update the Kubernetes secret** in place (the Hub expects a base64-encoded PEM inside the secret value): ```bash kubectl -n lunar create secret generic lunar-github-app \ --from-literal=private-key="$(base64 < path/to/new-key.pem | tr -d '\n')" \ --dry-run=client -o yaml | kubectl apply -f - ``` 3. **Roll the Hub** — `kubectl -n lunar rollout restart deployment/lunar-hub`. 4. **Verify** webhooks still deliver (GitHub → any repo → **Settings → Webhooks → Recent Deliveries**). 5. **Delete the old key** in GitHub. {% endtab %} {% tab title="GitLab" %} **Group access token (`lunar-gitlab-token`).** Revoking the old token takes effect immediately, so revoke it last: 1. **Create the replacement token** in GitLab (group → **Settings → Access tokens**), with the same scope and role as the original. 2. **Update the secret in place:** ```bash kubectl -n lunar create secret generic lunar-gitlab-token \ --from-literal=token='' \ --dry-run=client -o yaml | kubectl apply -f - ``` 3. **Roll the Hub** — `kubectl -n lunar rollout restart deployment/lunar-hub`. 4. **Revoke the old token** in GitLab. {% hint style="warning" %} GitLab access tokens expire on a fixed date and nothing renews them. When one lapses, Lunar silently stops posting results and reacting to webhooks for that group. Put the expiry date in a calendar and rotate ahead of it. {% endhint %} {% endtab %} {% endtabs %} ### Webhook secrets The chart generates one of these per platform. {% hint style="danger" %} **Rotating a webhook secret is not an online operation.** Lunar stamps the secret onto a hook when it **creates** it, and never updates an existing one — re-registration only reconciles the hook's event subscriptions and its enabled state. Changing the secret therefore breaks delivery on every repository that already has a hook, and neither `lunar hub pull` nor the reconcile loop will repair it. Each hook has to be deleted so Lunar recreates it. Rotate only when you have to, and plan for the whole fleet to be re-registered in one go. On GitLab the window is not free: failing deliveries get the hook [auto-disabled](#webhook-health), permanently after 40 consecutive failures. {% endhint %} The procedure is the same on both platforms: 1. **Replace the secret and restart.** Until the Hub restarts it keeps validating against the old value, and until it does, it also stamps the old value onto any hook it creates. 2. **Delete Lunar's hook on every affected repository**, via your platform's API. Lunar's hook is the one pointing at your Hub's `/webhooks/` URL. 3. **Let Lunar recreate them.** Recreation happens on the next repo sync, which is gated by the webhook freshness record: at worst `HUB_WEBHOOK_FRESHNESS` (12h by default) after the delete. New hooks carry the new secret. 4. **Confirm deliveries are succeeding again** before considering the rotation done — see [Webhook health](#webhook-health). {% tabs %} {% tab title="GitHub" %} ```bash kubectl -n lunar delete secret lunar-github-webhook helm upgrade lunar earthly/lunar -n lunar -f values.yaml kubectl -n lunar rollout restart deployment/lunar-hub ``` Then delete the hooks so they are recreated with the new secret. Between the restart and the recreation, GitHub deliveries fail signature validation and the Hub rejects them with `401`. GitHub does not disable a hook for failing, so nothing is lost permanently — but nothing reaches Lunar either, and the failures are only visible under **Settings → Webhooks → Recent Deliveries**. To rotate a per-app `webhook_secret` in `HUB_GITHUB_APPS`, edit that value instead of deleting the chart secret, then `helm upgrade` and restart. The hooks still have to be deleted and recreated. {% endtab %} {% tab title="GitLab" %} ```bash kubectl -n lunar delete secret lunar-gitlab-webhook helm upgrade lunar earthly/lunar -n lunar -f values.yaml kubectl -n lunar rollout restart deployment/lunar-hub ``` Then delete the hooks so they are recreated with the new secret. Do not leave projects sitting on the old secret. GitLab disables a hook after four consecutive failures and **permanently after forty**, and Lunar's automatic re-enable will not help while the secret is still wrong — it re-enables the hook, the next delivery fails, and the count keeps climbing. Delete and recreate promptly rather than waiting out the sync. To rotate a per-group `webhook_secret` in `HUB_GITLAB_TOKENS`, edit that value instead of deleting the chart secret, then `helm upgrade` and restart. Doing this one group at a time keeps each blast radius to a single group. {% endtab %} {% endtabs %} ### Database password (`lunar-db`) Standard Kubernetes secret rotation. If your Postgres supports multiple active passwords (e.g. RDS password grace), rotate without downtime: 1. Set the new password in Postgres (keeping the old one active). 2. Update the `lunar-db` secret. 3. Restart the Hub. 4. Revoke the old password. If your Postgres doesn't support multiple active passwords, expect \~1 minute of Hub unavailability during the rotation. ### Other secrets A few secrets don't get their own recipe above: | Secret | Managed by | Consumed by | | -------------------------------------------------------------------------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------- | | `-grafana-admin` | Chart | The bundled Grafana + the Hub (only in `grafana.mode: chart`) | | `-grafana-db` — read-only `grafana_user` role | Chart | Grafana datasource — whenever dashboards deploy (bundled **or** your own) | | Your own Grafana login (`HUB_GRAFANA_TOKEN`, or `HUB_GRAFANA_USER` + `HUB_GRAFANA_PASSWORD`) | You | Hub — only with your own Grafana ([Option B/C](/install/lunar-hub/self-hosted/install-walkthrough.md#grafana)) | | Hub licence JWT (example `lunar-hub-licence`) | You | Hub | | Per-scope runtime secrets (`hub.secrets.{collector,cataloger,policy}.secretName`) | You | Hub | The rotation shape follows the same split as the sections above: * **Chart-managed** (Grafana admin): `kubectl delete secret ` → `helm upgrade` to regenerate → restart the consumer deployment. This does **not** apply to `-grafana-db` — treat it as immutable after install. * **User-managed** (hub licence JWT, runtime secrets, your own Grafana credentials): update the secret in place (`kubectl apply` or `kubectl create --dry-run=client -o yaml | kubectl apply -f -`) → restart the consumer(s). After rotating your own Grafana credentials, restart the Hub so it picks up the new value — the next `helm upgrade` re-runs the chart's dashboard-deploy Job with it. To find what consumes an arbitrary secret: ```bash kubectl -n lunar get deployments -o yaml | grep -B2 '' ``` ## Webhook health Lunar registers a webhook on every tracked repo and re-verifies it periodically, repairing a hook that exists but has stopped delivering: * **GitLab** auto-disables a hook after 4 consecutive delivery failures (self-heals after a backoff) and **permanently after 40** — which a sustained problem, like a wrongly configured webhook secret or a weeks-long Hub outage, will eventually reach. Lunar detects the disabled state and re-enables the hook by sending a test request. * **GitHub** never disables a hook for failing, but a repo admin can untick **Active** in the repo's webhook settings. Lunar detects the inactive hook and switches it back on. Repair happens automatically on the periodic re-sync, at worst one freshness window (`HUB_WEBHOOK_FRESHNESS`, default 12h) plus one sync pass after the hook went dead. It only sticks if the underlying cause is fixed — a GitLab hook whose deliveries keep failing (e.g. a still-wrong webhook secret) will just be disabled again. Events that fired while the hook was dead are not replayed; neither forge redelivers them. ## Observability ### Logs The Hub and Operator log in structured JSON by default (`logging.format: json`). Log level defaults to `info`; raise to `debug` temporarily by editing `values.yaml` and running `helm upgrade`. The Hub and Operator don't ship logs anywhere themselves — they stream to stdout. To get them into your log aggregator, point a cluster-level log shipper (Fluent Bit, Vector, Datadog Agent, etc.) at the `lunar` namespace. ### Telemetry to Earthly Hub tenant identity and telemetry destinations come from the signed licence JWT, not from `HUB_TENANT_ID`, `HUB_ELASTIC_*`, or `HUB_OTEL_*` environment variables. The chart handles the mount and `HUB_LICENCE_FILE` wiring for you. Keep (or override) the licence secret reference in values: ```yaml hub: licence: secretName: lunar-hub-licence secretKey: hub-licence.jwt # optional; defaults to /var/run/secrets/lunar/hub-licence.jwt # filePath: /var/run/secrets/lunar/hub-licence.jwt ``` Rotation flow: * Request a new JWT from Earthly. * Update the Kubernetes secret value (`hub-licence.jwt`) in place. * The Hub re-validates every 5 minutes and also enforces the exact `exp` boundary. ### Metrics The Hub exports request-duration histograms (HTTP and gRPC) via OTLP when the active licence includes `telemetry.otel.endpoint` and `telemetry.otel.token`. If `telemetry.otel` is omitted, the Hub falls back to writing metrics to a local temp file. Secure (TLS) OTLP is not implemented yet, so OTLP export runs insecure today. {% hint style="info" %} The Hub does not export distributed traces today — only metrics. Trace support is on the roadmap. {% endhint %} ### Aggregating metrics across replicas With multiple replicas, be deliberate about how dashboards and alerts aggregate the Hub's job-queue metrics. Queue-depth **gauges** (e.g. `river_jobs`) read from the queue tables in Postgres, which are **shared** by every replica, so each replica reports the same global count — aggregate with **`max`** (or `avg`), never `sum`, which multiplies the backlog by the replica count. Per-replica **counters** are the opposite: `river_insert_count_total` / `river_work_count_total` count work done by *each* replica, so `sum(rate(...))` is the correct cluster-wide throughput. ### Diagnostics bundle The Hub can produce a diagnostics bundle (Postgres queue state, recent error logs, slow-query stats) for support requests. Generate one with: ```bash lunar hub get-logs ``` This collects `kubectl` logs from the Hub pods plus Postgres telemetry from the Hub, then writes a `lunar-hub-logs-.tar.gz` in the current directory and prints its path. Run it from a machine with `kubectl` access to the Hub's cluster and your Lunar CLI configured to reach the Hub. Common flags: * `-n`, `--namespace` — Kubernetes namespace to read Hub pods from (defaults to the current kubectl context's namespace). * `-o`, `--output` — path to write the bundle to (defaults to `lunar-hub-logs-.tar.gz` in the current directory). * `--tail` — log lines to capture per container (default: `1000`). The bundle gathers extra Postgres telemetry when the `pg_stat_statements` extension is enabled — without it the bundle still works, just with less query-performance data. To enable the extension: * **Amazon RDS / Aurora:** add `pg_stat_statements` to `shared_preload_libraries` in the DB parameter group, then reboot the instance. * **Self-managed:** add `pg_stat_statements` to `shared_preload_libraries` in `postgresql.conf` and restart, then `CREATE EXTENSION pg_stat_statements;` as a superuser. The Hub does not require this extension — it's only used by the diagnostics path. ## Scaling The Hub is stateless — all durable state lives in Postgres and S3 — so it scales **horizontally**: raise `hub.replicaCount` and the replicas coordinate through Postgres (leader election, advisory locks) with no peer-to-peer wiring. Size the connection pool against your Postgres `max_connections` first — see [Replicas and the Postgres connection budget](/install/lunar-hub/self-hosted/install-walkthrough.md#replicas-and-the-postgres-connection-budget). Add a PodDisruptionBudget (`hub.podDisruptionBudget.enabled: true`) so node drains and rollouts keep a quorum. Vertical scaling still helps a single busy replica: CPU, memory, and the Postgres pool via `hub.db.maxOpenConns` / `hub.db.maxPoolConns`. Run throughput scales independently — raise `operator.maxConcurrent` and size run pod resources accordingly. See [`operator.scriptContainerSpec*`](https://github.com/earthly/charts/blob/main/README.md#values-reference) in the chart README. ### Leader election and periodic jobs Exactly one replica is the elected **leader** at a time; it runs the periodic and maintenance jobs (cleanup, reconcile, backfill). Jobs marked `RunOnStart` fire **immediately whenever leadership is (re)acquired** — not just at process boot — so a failover re-runs them on the new leader. Their handlers must be **idempotent**; the current `RunOnStart` jobs already are. Don't add one whose effect isn't safe to repeat on an arbitrary leadership flip. ## Prioritizing core services over script pods By default the Operator creates script pods (collectors, policies, catalogers) at the cluster's default priority — the same priority as the Hub, Operator, and Grafana. When those script pods share a node with the core services, a burst of runs can drive the node into memory pressure and the kubelet may evict a *core* pod when it should be shedding an ephemeral script instead. This matters in two setups: * **Single-namespace installs** — the Operator runs scripts in the same namespace (and usually the same node group) as the Hub, so core and ephemeral workloads compete for the same memory. * **Shared scratch namespaces or node groups** — scripts land on nodes shared with other workloads, and you want Lunar's ephemeral pods to yield first under contention. `operator.scriptPodPriorityClassName` (chart `>= 2.3.0`) sets the [PriorityClass](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/) on every script pod the Operator creates. Point it at a class whose value sits *below* your core services, and the kubelet evicts scripts first under pressure — the scheduler also won't preempt a higher-priority pod to make room for a script. ### Setup 1. **Create the PriorityClass.** The chart references it by name but does not create it. ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: lunar-script value: -10 # negative → below the default (0) your core pods use globalDefault: false description: Ephemeral Lunar script pods; evicted before core services. ``` ```bash kubectl apply -f lunar-script-priorityclass.yaml ``` 2. **Point the Operator at it** in `values.yaml`: ```yaml operator: scriptPodPriorityClassName: lunar-script ``` 3. **Upgrade and restart the Operator** — only script pods created *after* the restart inherit the class: ```bash helm upgrade lunar earthly/lunar -n lunar -f values.yaml kubectl -n lunar rollout restart deployment/lunar-operator ``` Leave the Hub, Operator, and Grafana on the cluster default priority — don't set a class on them. The gap between their `0` and the script class's negative value is what orders eviction; you lower scripts rather than raise core services. In a shared scratch node group, pick a value below whatever neighbouring workloads run at so Lunar's scripts are the first thing to go. {% hint style="warning" %} PriorityClasses are **cluster-scoped**, and a pod that references one that doesn't exist is rejected at admission — so the Operator will fail to launch scripts until the class is applied. Create the class *before* (or in the same change as) the `helm upgrade`, and delete it only once no workload references it. {% endhint %} {% hint style="info" %} Priority orders *who gets evicted first* — it is not a resource cap. Keep your script pod requests/limits set (see [`operator.scriptContainerSpec*`](https://github.com/earthly/charts/blob/main/README.md#values-reference) in the chart README) so a single run can't consume a whole node before the kubelet reacts. Priority decides the order; requests and limits decide the ceiling. {% endhint %} ## Operating the GitLab merge gate If you run [the merge gate](/install/git-platforms/gitlab.md#merge-gate) on GitLab Ultimate, two of its properties are operational rather than install-time: * **It is fail-closed.** A Hub outage leaves gated merge requests blocked. That is the correct default for a compliance control, but it means a Lunar incident becomes a merge freeze. * **The lever is on the GitLab side.** To unblock merges during an incident, a project Owner disables **Status checks must succeed** in the project's merge-request settings. Lunar re-enables it on the next sync, so this buys time rather than being an opt-out. A project that should be permanently ungated belongs out of Lunar's configuration instead. Individual blocked merge requests do not need an incident response — an authorized engineer can comment `/lunar bypass: ` on the merge request, which is recorded and applies to that commit only. ## Backup and disaster recovery Lunar's authoritative state lives in the systems you already back up: * **Postgres** — use your existing Postgres backup process. Everything the Hub cares about (components, policies, run history, queue state) is here. * **S3 buckets** — enable object versioning on both buckets and, if compliance requires, cross-region replication. Lost resource archives cause re-building the cache of catalogers, collectors, and policies; lost log archives mean lost history. Neither impedes Hub operation. * **Hub local scratch** — each Hub pod's `/var/lib/lunar` is an ephemeral `emptyDir` (re-extracted runtimes and cached bundles; the authoritative copy is in S3). It is per-pod and regenerable, so there's nothing to back up. * **Kubernetes secrets** — keep your Git platform credential (the GitHub App PEM or the GitLab group access token), DB credentials, auth token, and webhook secret in your organization's secret-management system of record. Losing them means re-provisioning. ## Uninstalling ```bash helm uninstall lunar --namespace lunar ``` Helm removes all resources the chart created: the Hub and operator deployments, services, ingress, RBAC, and service accounts. **Not removed automatically:** * The `lunar` namespace itself (and, if used, the Operator's execution namespace). * Kubernetes secrets you created manually (e.g. `lunar-db`, `lunar-github-app`, `lunar-gitlab-token`, `lunar-hub-licence`). * **Anything Lunar provisioned on your GitLab projects** — webhooks, the `Earthly Lunar` status check, and the **Status checks must succeed** setting. Uninstalling the Hub does not undo them, and with the Hub gone nothing answers the status check, so gated merge requests stay blocked. Before uninstalling, disable that setting on the affected projects. * Chart-managed secrets (`-auth-token`, `-github-webhook`, `-gitlab-webhook`, `-grafana-admin`). These carry `helm.sh/resource-policy: keep` so a re-install reuses the same values — by design. Delete them explicitly if you want fresh credentials on the next install. * Your Postgres database, S3 buckets, or their contents. * The GitHub App. Webhooks previously registered by this Hub are tagged with the Hub's instance ID (`earthly-lunar-` as a URL fragment); they remain on your repos until you remove them manually or install a fresh Hub with the same tenant ID, which will clean up its own stale webhooks on next config pull. To fully tear down: ```bash helm uninstall lunar --namespace lunar kubectl -n lunar delete secret lunar-db lunar-github-app lunar-gitlab-token \ lunar-hub-licence \ lunar-auth-token lunar-github-webhook lunar-gitlab-webhook lunar-grafana-admin \ --ignore-not-found kubectl delete namespace lunar # Externally: # - Drop the Postgres database # - Delete or empty the S3 buckets # - Uninstall the GitHub App from your organization # - Revoke the GitLab group access token, and remove the status check and # "Status checks must succeed" setting from any gated projects ``` ## Getting help * [Chart values reference](https://github.com/earthly/charts/blob/main/README.md#values-reference) * [Chart source](https://github.com/earthly/charts) * [Lunar source](https://github.com/earthly/lunar) * For enterprise onboarding or production sizing guidance, [contact the Earthly team](https://earthly.dev/earthly-lunar/demo). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs-lunar.earthly.dev/install/lunar-hub/self-hosted/day-2-operations.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.