Ctrlk
HomeGuardrailsIntegrationsBlogAutomate Now ◉Get a Demo
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Day-2 Operations

This page covers what you need to run Lunar Hub after the initial install: upgrades, secret rotation, observability, and uninstall. For the per-version upgrade notes (what values changed, what to migrate), see the chart README's Upgrading section.

Upgrading

helm repo update
helm upgrade lunar earthly/lunar \
  --namespace lunar \
  -f values.yaml

Migrations run once per release as a pre-rollout Job (lunar-hub-migrate), installed via a Helm pre-install/pre-upgrade hook that completes — and gates the rollout — before any new Hub pod is created. Serving pods then assert the schema is current at boot and refuse to start if it's behind, which also keeps scaled-up and restarted pods safe (they don't re-run the Job). Migrations are forward-only and an interrupted one is safe to retry.

Before every upgrade:

  1. Back up Postgres. Migrations are forward-only. If you need to roll back, roll the database back.

  2. Preview the chart changes. The helm-diff plugin is worth installing — helm diff upgrade lunar earthly/lunar -f values.yaml shows exactly what Kubernetes resources will change.

  3. Read the version-specific notes in the chart README. Minor version bumps occasionally require values changes.

  4. Pin your new image tags in values.yaml. Don't rely on the chart's default tags.

Migrations are forward-only — back up Postgres first (step 1). Author schema changes expand/contract (add first, drop in a later release) so the common upgrade stays zero-downtime: the migrate Job runs while the old-version pods keep serving, then Helm rolls the Deployment. Reserve a maintenance window only for an unavoidable breaking change, and confirm the budget with stakeholders — a heavy migration can take tens of minutes, during which the old version keeps serving.

With multiple replicas and a PodDisruptionBudget, Helm's rolling update replaces Hub pods a few at a time while the rest keep serving, so a routine upgrade has no API-unavailability window.

Rotating secrets

Lunar's secrets split into two categories — user-managed (you create and rotate them) and chart-managed (auto-generated on first install, preserved across upgrades via helm.sh/resource-policy: keep). See Required secrets in the chart README for the full list and naming rules.

Hub auth token (<release>-auth-token)

Used by the CLI and CI tracers to authenticate to the Hub. The Hub accepts a single token today — rotation requires a coordinated cutover. To force a regeneration, delete the secret and re-run helm upgrade (the chart will generate a fresh token):

Both the Hub and the Operator read this token from the same secret (the Operator uses it as OPERATOR_HUB_TOKEN to call the Hub), so restart both. Retrieve the new token with kubectl -n lunar get secret lunar-auth-token -o jsonpath='{.data.token}' | base64 -d, then update every CI tracer's LUNAR_HUB_TOKEN and every developer's CLI config. Active builds authenticated with the old token will fail mid-run and need to be retried. Dual-token support is on the roadmap.

To pin to an externally-managed token instead, set hub.auth.secretName in values to a secret you control — the chart will consume it instead of generating its own.

Git platform credentials

Both are read once at boot, so a rolling restart is always required after updating the secret.

GitHub App private key (lunar-github-app). GitHub allows multiple active keys per App, so this rotates with zero downtime:

  1. Generate a new key in GitHub → Apps → your App → Private keys → Generate a private key. Download the new PEM.

  2. Update the Kubernetes secret in place (the Hub expects a base64-encoded PEM inside the secret value):

  3. Roll the Hub — kubectl -n lunar rollout restart deployment/lunar-hub.

  4. Verify webhooks still deliver (GitHub → any repo → Settings → Webhooks → Recent Deliveries).

  5. Delete the old key in GitHub.

Group access token (lunar-gitlab-token). Revoking the old token takes effect immediately, so revoke it last:

  1. Create the replacement token in GitLab (group → Settings → Access tokens), with the same scope and role as the original.

  2. Update the secret in place:

  3. Roll the Hub — kubectl -n lunar rollout restart deployment/lunar-hub.

  4. Revoke the old token in GitLab.

GitLab access tokens expire on a fixed date and nothing renews them. When one lapses, Lunar silently stops posting results and reacting to webhooks for that group. Put the expiry date in a calendar and rotate ahead of it.

Webhook secrets

The chart generates one of these per platform.

Rotating a webhook secret is not an online operation. Lunar stamps the secret onto a hook when it creates it, and never updates an existing one — re-registration only reconciles the hook's event subscriptions and its enabled state. Changing the secret therefore breaks delivery on every repository that already has a hook, and neither lunar hub pull nor the reconcile loop will repair it. Each hook has to be deleted so Lunar recreates it.

Rotate only when you have to, and plan for the whole fleet to be re-registered in one go. On GitLab the window is not free: failing deliveries get the hook auto-disabled, permanently after 40 consecutive failures.

The procedure is the same on both platforms:

  1. Replace the secret and restart. Until the Hub restarts it keeps validating against the old value, and until it does, it also stamps the old value onto any hook it creates.

  2. Delete Lunar's hook on every affected repository, via your platform's API. Lunar's hook is the one pointing at your Hub's /webhooks/<platform> URL.

  3. Let Lunar recreate them. Recreation happens on the next repo sync, which is gated by the webhook freshness record: at worst HUB_WEBHOOK_FRESHNESS (12h by default) after the delete. New hooks carry the new secret.

  4. Confirm deliveries are succeeding again before considering the rotation done — see Webhook health.

Then delete the hooks so they are recreated with the new secret.

Between the restart and the recreation, GitHub deliveries fail signature validation and the Hub rejects them with 401. GitHub does not disable a hook for failing, so nothing is lost permanently — but nothing reaches Lunar either, and the failures are only visible under Settings → Webhooks → Recent Deliveries.

To rotate a per-app webhook_secret in HUB_GITHUB_APPS, edit that value instead of deleting the chart secret, then helm upgrade and restart. The hooks still have to be deleted and recreated.

Then delete the hooks so they are recreated with the new secret.

Do not leave projects sitting on the old secret. GitLab disables a hook after four consecutive failures and permanently after forty, and Lunar's automatic re-enable will not help while the secret is still wrong — it re-enables the hook, the next delivery fails, and the count keeps climbing. Delete and recreate promptly rather than waiting out the sync.

To rotate a per-group webhook_secret in HUB_GITLAB_TOKENS, edit that value instead of deleting the chart secret, then helm upgrade and restart. Doing this one group at a time keeps each blast radius to a single group.

Database password (lunar-db)

Standard Kubernetes secret rotation. If your Postgres supports multiple active passwords (e.g. RDS password grace), rotate without downtime:

  1. Set the new password in Postgres (keeping the old one active).

  2. Update the lunar-db secret.

  3. Restart the Hub.

  4. Revoke the old password.

If your Postgres doesn't support multiple active passwords, expect ~1 minute of Hub unavailability during the rotation.

Other secrets

A few secrets don't get their own recipe above:

Secret
Managed by
Consumed by

<release>-grafana-admin

Chart

The bundled Grafana + the Hub (only in grafana.mode: chart)

<release>-grafana-db — read-only grafana_user role

Chart

Grafana datasource — whenever dashboards deploy (bundled or your own)

Your own Grafana login (HUB_GRAFANA_TOKEN, or HUB_GRAFANA_USER + HUB_GRAFANA_PASSWORD)

You

Hub — only with your own Grafana (Option B/C)

Hub licence JWT (example lunar-hub-licence)

You

Hub

Per-scope runtime secrets (hub.secrets.{collector,cataloger,policy}.secretName)

You

Hub

The rotation shape follows the same split as the sections above:

  • Chart-managed (Grafana admin): kubectl delete secret <name> → helm upgrade to regenerate → restart the consumer deployment. This does not apply to <release>-grafana-db — treat it as immutable after install.

  • User-managed (hub licence JWT, runtime secrets, your own Grafana credentials): update the secret in place (kubectl apply or kubectl create --dry-run=client -o yaml | kubectl apply -f -) → restart the consumer(s). After rotating your own Grafana credentials, restart the Hub so it picks up the new value — the next helm upgrade re-runs the chart's dashboard-deploy Job with it.

To find what consumes an arbitrary secret:

Webhook health

Lunar registers a webhook on every tracked repo and re-verifies it periodically, repairing a hook that exists but has stopped delivering:

  • GitLab auto-disables a hook after 4 consecutive delivery failures (self-heals after a backoff) and permanently after 40 — which a sustained problem, like a wrongly configured webhook secret or a weeks-long Hub outage, will eventually reach. Lunar detects the disabled state and re-enables the hook by sending a test request.

  • GitHub never disables a hook for failing, but a repo admin can untick Active in the repo's webhook settings. Lunar detects the inactive hook and switches it back on.

Repair happens automatically on the periodic re-sync, at worst one freshness window (HUB_WEBHOOK_FRESHNESS, default 12h) plus one sync pass after the hook went dead. It only sticks if the underlying cause is fixed — a GitLab hook whose deliveries keep failing (e.g. a still-wrong webhook secret) will just be disabled again. Events that fired while the hook was dead are not replayed; neither forge redelivers them.

Observability

Logs

The Hub and Operator log in structured JSON by default (logging.format: json). Log level defaults to info; raise to debug temporarily by editing values.yaml and running helm upgrade.

The Hub and Operator don't ship logs anywhere themselves — they stream to stdout. To get them into your log aggregator, point a cluster-level log shipper (Fluent Bit, Vector, Datadog Agent, etc.) at the lunar namespace.

Telemetry to Earthly

Hub tenant identity and telemetry destinations come from the signed licence JWT, not from HUB_TENANT_ID, HUB_ELASTIC_*, or HUB_OTEL_* environment variables.

The chart handles the mount and HUB_LICENCE_FILE wiring for you. Keep (or override) the licence secret reference in values:

Rotation flow:

  • Request a new JWT from Earthly.

  • Update the Kubernetes secret value (hub-licence.jwt) in place.

  • The Hub re-validates every 5 minutes and also enforces the exact exp boundary.

Metrics

The Hub exports request-duration histograms (HTTP and gRPC) via OTLP when the active licence includes telemetry.otel.endpoint and telemetry.otel.token.

If telemetry.otel is omitted, the Hub falls back to writing metrics to a local temp file.

Secure (TLS) OTLP is not implemented yet, so OTLP export runs insecure today.

The Hub does not export distributed traces today — only metrics. Trace support is on the roadmap.

Aggregating metrics across replicas

With multiple replicas, be deliberate about how dashboards and alerts aggregate the Hub's job-queue metrics. Queue-depth gauges (e.g. river_jobs) read from the queue tables in Postgres, which are shared by every replica, so each replica reports the same global count — aggregate with max (or avg), never sum, which multiplies the backlog by the replica count. Per-replica counters are the opposite: river_insert_count_total / river_work_count_total count work done by each replica, so sum(rate(...)) is the correct cluster-wide throughput.

Diagnostics bundle

The Hub can produce a diagnostics bundle (Postgres queue state, recent error logs, slow-query stats) for support requests. Generate one with:

This collects kubectl logs from the Hub pods plus Postgres telemetry from the Hub, then writes a lunar-hub-logs-<timestamp>.tar.gz in the current directory and prints its path. Run it from a machine with kubectl access to the Hub's cluster and your Lunar CLI configured to reach the Hub. Common flags:

  • -n, --namespace — Kubernetes namespace to read Hub pods from (defaults to the current kubectl context's namespace).

  • -o, --output — path to write the bundle to (defaults to lunar-hub-logs-<timestamp>.tar.gz in the current directory).

  • --tail — log lines to capture per container (default: 1000).

The bundle gathers extra Postgres telemetry when the pg_stat_statements extension is enabled — without it the bundle still works, just with less query-performance data. To enable the extension:

  • Amazon RDS / Aurora: add pg_stat_statements to shared_preload_libraries in the DB parameter group, then reboot the instance.

  • Self-managed: add pg_stat_statements to shared_preload_libraries in postgresql.conf and restart, then CREATE EXTENSION pg_stat_statements; as a superuser.

The Hub does not require this extension — it's only used by the diagnostics path.

Scaling

The Hub is stateless — all durable state lives in Postgres and S3 — so it scales horizontally: raise hub.replicaCount and the replicas coordinate through Postgres (leader election, advisory locks) with no peer-to-peer wiring. Size the connection pool against your Postgres max_connections first — see Replicas and the Postgres connection budget. Add a PodDisruptionBudget (hub.podDisruptionBudget.enabled: true) so node drains and rollouts keep a quorum.

Vertical scaling still helps a single busy replica: CPU, memory, and the Postgres pool via hub.db.maxOpenConns / hub.db.maxPoolConns.

Run throughput scales independently — raise operator.maxConcurrent and size run pod resources accordingly. See operator.scriptContainerSpec* in the chart README.

Leader election and periodic jobs

Exactly one replica is the elected leader at a time; it runs the periodic and maintenance jobs (cleanup, reconcile, backfill). Jobs marked RunOnStart fire immediately whenever leadership is (re)acquired — not just at process boot — so a failover re-runs them on the new leader. Their handlers must be idempotent; the current RunOnStart jobs already are. Don't add one whose effect isn't safe to repeat on an arbitrary leadership flip.

Prioritizing core services over script pods

By default the Operator creates script pods (collectors, policies, catalogers) at the cluster's default priority — the same priority as the Hub, Operator, and Grafana. When those script pods share a node with the core services, a burst of runs can drive the node into memory pressure and the kubelet may evict a core pod when it should be shedding an ephemeral script instead.

This matters in two setups:

  • Single-namespace installs — the Operator runs scripts in the same namespace (and usually the same node group) as the Hub, so core and ephemeral workloads compete for the same memory.

  • Shared scratch namespaces or node groups — scripts land on nodes shared with other workloads, and you want Lunar's ephemeral pods to yield first under contention.

operator.scriptPodPriorityClassName (chart >= 2.3.0) sets the PriorityClass on every script pod the Operator creates. Point it at a class whose value sits below your core services, and the kubelet evicts scripts first under pressure — the scheduler also won't preempt a higher-priority pod to make room for a script.

Setup

  1. Create the PriorityClass. The chart references it by name but does not create it.

  2. Point the Operator at it in values.yaml:

  3. Upgrade and restart the Operator — only script pods created after the restart inherit the class:

Leave the Hub, Operator, and Grafana on the cluster default priority — don't set a class on them. The gap between their 0 and the script class's negative value is what orders eviction; you lower scripts rather than raise core services. In a shared scratch node group, pick a value below whatever neighbouring workloads run at so Lunar's scripts are the first thing to go.

PriorityClasses are cluster-scoped, and a pod that references one that doesn't exist is rejected at admission — so the Operator will fail to launch scripts until the class is applied. Create the class before (or in the same change as) the helm upgrade, and delete it only once no workload references it.

Priority orders who gets evicted first — it is not a resource cap. Keep your script pod requests/limits set (see operator.scriptContainerSpec* in the chart README) so a single run can't consume a whole node before the kubelet reacts. Priority decides the order; requests and limits decide the ceiling.

Operating the GitLab merge gate

If you run the merge gate on GitLab Ultimate, two of its properties are operational rather than install-time:

  • It is fail-closed. A Hub outage leaves gated merge requests blocked. That is the correct default for a compliance control, but it means a Lunar incident becomes a merge freeze.

  • The lever is on the GitLab side. To unblock merges during an incident, a project Owner disables Status checks must succeed in the project's merge-request settings. Lunar re-enables it on the next sync, so this buys time rather than being an opt-out. A project that should be permanently ungated belongs out of Lunar's configuration instead.

Individual blocked merge requests do not need an incident response — an authorized engineer can comment lunar bypass: <reason> on the merge request, which is recorded and applies to that commit only.

Backup and disaster recovery

Lunar's authoritative state lives in the systems you already back up:

  • Postgres — use your existing Postgres backup process. Everything the Hub cares about (components, policies, run history, queue state) is here.

  • S3 buckets — enable object versioning on both buckets and, if compliance requires, cross-region replication. Lost resource archives cause re-building the cache of catalogers, collectors, and policies; lost log archives mean lost history. Neither impedes Hub operation.

  • Hub local scratch — each Hub pod's /var/lib/lunar is an ephemeral emptyDir (re-extracted runtimes and cached bundles; the authoritative copy is in S3). It is per-pod and regenerable, so there's nothing to back up.

  • Kubernetes secrets — keep your Git platform credential (the GitHub App PEM or the GitLab group access token), DB credentials, auth token, and webhook secret in your organization's secret-management system of record. Losing them means re-provisioning.

Uninstalling

Helm removes all resources the chart created: the Hub and operator deployments, services, ingress, RBAC, and service accounts.

Not removed automatically:

  • The lunar namespace itself (and, if used, the Operator's execution namespace).

  • Kubernetes secrets you created manually (e.g. lunar-db, lunar-github-app, lunar-gitlab-token, lunar-hub-licence).

  • Anything Lunar provisioned on your GitLab projects — webhooks, the Earthly Lunar status check, and the Status checks must succeed setting. Uninstalling the Hub does not undo them, and with the Hub gone nothing answers the status check, so gated merge requests stay blocked. Before uninstalling, disable that setting on the affected projects.

  • Chart-managed secrets (<release>-auth-token, <release>-github-webhook, <release>-gitlab-webhook, <release>-grafana-admin). These carry helm.sh/resource-policy: keep so a re-install reuses the same values — by design. Delete them explicitly if you want fresh credentials on the next install.

  • Your Postgres database, S3 buckets, or their contents.

  • The GitHub App. Webhooks previously registered by this Hub are tagged with the Hub's instance ID (earthly-lunar-<tenant-id-from-licence> as a URL fragment); they remain on your repos until you remove them manually or install a fresh Hub with the same tenant ID, which will clean up its own stale webhooks on next config pull.

To fully tear down:

Getting help

PreviousSync ConfigNextLunar Dedicated

Last updated