Ctrlk
HomeGuardrailsIntegrationsBlogAutomate Now ◉Get a Demo
For the complete documentation index, see llms.txt. This page is also available as Markdown.

2026

2026-09-02

Helm chart version: 3.20.0 Lunar images version: 3.20.0 Product updates included: 2026-09-02

Product updates

Breaking changes

HUB_SQLAPI_PASSWORD in hub.extraEnv now fails the render

The chart owns HUB_SQLAPI_PASSWORD as of this release, and an install that still sets it under hub.extraEnv fails to render, with an error naming the values to move it to. Leaving both copies in place would let Kubernetes accept the duplicate and pick a winner by template ordering rather than by intent, and a GitOps install on server-side apply would reject the object outright on the duplicate key. The render aborts before Helm touches the cluster, so nothing is half-applied and there is nothing to roll back.

To keep the existing password, set mode: secret with secretName and passwordKey pointing at the same Secret your extraEnv entry referenced: nothing rotates, and connection strings already in use keep working. If you supplied the password inline as value: rather than through a secretKeyRef, create a Secret holding it first — the chart takes credentials only by reference, never as a literal value.

To hand the password to the chart instead, drop the extraEnv entry and let the default apply. The chart generates its own Secret under a different name, so the old one is left unreferenced and can be deleted afterwards as cleanup. The password rotates, so re-fetch it with lunar sql connection-string and update any client holding the old one.

Do not delete the old Secret first: deleting it destroys the only copy of the password, and with it the option of keeping it. Change the values first and clean up afterwards.

  • The Hub no longer reads HUB_SCOPED_MATERIALIZE_ENABLED or HUB_ASSOCIATE_COMPONENT_DEDUP_ENABLED: both are now ignored wherever they are set, and catalog materialization is always scoped to what a change touched. An installation that set HUB_SCOPED_MATERIALIZE_ENABLED=false to fall back to the whole-catalog rebuild now runs the scoped path regardless, because that rebuild is removed — the fallback was never a clean rollback in any case, since it served a catalog snapshot frozen at the moment scoped materialization took over. Remove both variables from hub.extraEnv so nothing in your values claims an effect it no longer has. See Scoped catalog materialization.

Migrations and upgrade notes

SQL API view definitions move to the materialized projections

Upgrading installs the materialized-projection definitions of the SQL API views. The lunar-hub-migrate Job renders them by default now, after checking that an existing installation's projections are present and populated; when that check does not pass — for example while a projection is still being filled — the Job leaves the previous view definitions in place, completes successfully, and tries again on the next deploy. A database with no Lunar migration history is bootstrapped straight onto the new definitions, since it has no history to catch up on.

To stay on, or return to, the previous definitions, set HUB_MAT_SERVING_ENABLED, HUB_MAT_POLICY_RUN_LATEST_ENABLED, or HUB_CHECKS_LATEST_SERVING_ENABLED to false under hub.extraEnv, which the migrate Job inherits, and redeploy. Each switch is independent and there is no data migration in either direction.

Because the views no longer derive recent rows at query time, the materialization write path is what keeps them current: turning it off with HUB_MAT_BACKGROUND_ENABLED=false while serving from the projections leaves the SQL API answering from a snapshot that stops advancing, without erroring. The Hub logs that as a staleness hazard.

See SQL API served from materialized tables.

Shared Postgres clusters must set hub.db.sqlapiPassword.mode: unmanaged before upgrading

If you pre-created sqlapi_user yourself because the Hub's database role cannot be granted cluster-wide CREATEROLE — the arrangement the self-hosted prerequisites describe for a shared Postgres cluster — set hub.db.sqlapiPassword.mode: unmanaged in your values before upgrading.

The new default is generate, which makes the migration issue ALTER ROLE sqlapi_user PASSWORD … against a role the Hub's database role has no authority over. Postgres refuses with permission denied to alter role, the SQL API code package is applied in a single transaction, and the migrate Job is a pre-upgrade hook, so the release aborts after the upgrade has already started. The chart cannot inspect the database role's privileges, so it cannot warn you.

Setting mode: unmanaged renders exactly what your install rendered before — no HUB_SQLAPI_PASSWORD in either the Hub Deployment or the migrate Job, and no ALTER ROLE — so nothing else about the install changes.

If you also run Grafana, unmanaged alone is not enough. grafana_user is created by the same mechanism with the same unguarded ALTER ROLE, and its password is chart-generated whenever grafana.mode is not off, with no equivalent opt-out. A shared-cluster install with Grafana enabled cannot complete a migration today, independently of this release; grafana.mode: off is the way through until that is addressed separately.

  • Upgrading runs a database migration that drops the two retired whole-catalog snapshot tables, hub.rel_catalog_json_items and hub.catalog_json. Nothing has written or read them since scoped catalog materialization became the default: the current catalog is derived from the live component and domain rows, and catalog history is served from the captured versions behind the SQL API catalog and catalog_latest views, so nothing a query or a dashboard can reach is lost and no operator action is required. Dropping the tables returns their space to the operating system straight away rather than leaving it allocated, which on a long-lived installation can be considerable — 14.6 GB on the installation this was sized against. Let the rollout finish promptly: a Hub pod from the previous release that is still running after the pre-rollout lunar-hub-migrate Job has applied the migration errors if it touches the dropped tables, for example on a retention sweep, and recovers once it is replaced by a pod from this release.

  • Setting HUB_REPO_SYNC_SWEEP_INTERVAL to 0 now stops repository reconciliation altogether, and the Hub starts without complaint. With the whole-catalog rebuild gone, nothing else re-enqueues a repository sync for a component that no catalog change has touched, so a disabled sweep leaves repository webhooks un-reasserted and commit and pull-request state unrefreshed on a settled installation. Leave the interval at its 15m default, or set another positive value, if you had turned the sweep off.

Security

  • The Lunar images upgrade Go dependencies carrying advisories flagged by software-composition analysis: golang.org/x/crypto to v0.55.0, resolving the critical CVE-2026-56854, and, for high-severity advisories, google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf, xDS RBAC and HTTP/2), github.com/labstack/echo/v4 to v4.15.3 (CVE-2026-55677), and golang.org/x/net to v0.57.0 (CVE-2026-46600); go.opentelemetry.io/otel moves to v1.44.0, github.com/klauspost/compress to v1.18.7, golang.org/x/sys to v0.47.0, golang.org/x/term to v0.45.0, and golang.org/x/text to v0.41.0 as part of the same upgrades. Upgrading the chart pulls the patched images; behavior is unchanged.

Features

Chart-managed password for the SQL API database role

The chart now supplies the password for sqlapi_user, the read-only Postgres role behind the connection string lunar sql connection-string vends, through the new hub.db.sqlapiPassword value. mode: generate, the default, creates a random 32-character alphanumeric password and keeps it across upgrades, so a connection string already handed to a BI tool or a saved psql invocation keeps working; the character set is deliberate, because the Hub builds the connection string without percent-encoding and a symbol would produce a URL clients cannot parse. mode: secret reads the password from a Secret you manage, named by secretName and passwordKey. mode: unmanaged renders no environment variable at all, for an install that pre-created the role and owns its credential.

There was no chart value for this before, and setting nothing did not produce an empty password: a fresh database got a role with no usable password verifier, unable to authenticate by any means, while the Hub went on vending a connection string with an empty password. Only databases created without a password were affected — an already-migrated database keeps the password it was first created with.

Such an install repairs itself on upgrade. The SQL API code package is re-applied in full on every migrate run, so the role's password is set to the value the chart supplies; there is no manual ALTER ROLE to run and nothing to clean up. With mode: unmanaged, a role you pre-created keeps the password you gave it. Under GitOps tooling that renders without reading the cluster — Argo CD, or helm template — generate cannot read an existing generated Secret back, so use mode: secret there, as with hub.auth.secretName.

  • The Hub's repository reconciliation concurrency is now configurable with HUB_MAX_WORKERS_REPO_SYNC, set through hub.extraEnv; it caps concurrent repository sync jobs per Hub replica, defaults to 5 — the previously hard-coded value, so an upgrade changes nothing — and 0 means unlimited, matching the other worker caps. A value below zero is rejected at startup. Lowering the cap spreads GitHub and GitLab API traffic over more time but does not reduce the total number of requests reconciliation makes, so watch the repo_sync queue backlog and its oldest available job age after lowering it: sustained growth means the cap is below the incoming work rate.

Improvements

  • An installation that lengthens the repository reconcile window with HUB_GITHUB_SYNC_RECONCILE_WINDOW now gets a startup warning when HUB_GITHUB_ETAG_CACHE_TTL (12 hours by default) is not longer than that window plus its 50% per-repository jitter, because cached GitHub responses then expire before a repository is next revisited and never reach the revalidation that would renew them. Raise the TTL above the reconcile window plus jitter; it bounds shared retention only, since each Hub replica keeps its in-process copy for at most 12 hours regardless. See GitHub conditional-request cache renewal.

  • The Hub now reports how much of its GitHub REST budget each kind of work consumes: every request that reaches GitHub is counted and timed, published as lunar.github.requests and lunar.github.request_duration over OTLP and as lunar_github_requests_total and lunar_github_request_duration_seconds on the /metrics scrape endpoint, alongside the existing per-installation rate-limit gauges that show how much budget remains. Both carry three fixed labels only — operation (the endpoint family, such as pr_files, repo_commits, check_runs, hooks, or issue_comments), source (pull_request, workflow_run, repo_sync, webhook_heal, or other), and outcome (success, not_modified, rate_limited, or error) — deliberately without owner, repository, pull-request, commit, URL, or delivery-ID labels, so cardinality stays bounded. Retries and pagination are counted as separate attempts because each one spends from the budget, and a conditional request that GitHub answers with 304 is recorded as not_modified even though the cache replays it as a 200. The Hub also logs one line per configured GitHub App pool at startup, giving the host, owner, and number of Apps in that pool, so available pool capacity can be confirmed without exposing credentials or installation IDs.

  • Data retention now ages out the cataloger delta rows in hub.catalog_json_items that links from the retired catalog snapshots used to pin in place. On an installation carrying data from before scoped catalog materialization those links held most aged delta rows out of every sweep — two thirds of the aged rows where this was measured — so the rows were never reclaimed however old they got. With the snapshot tables gone, an aged delta row is kept only while a live catalog contribution still references it, and is deleted otherwise.

  • The sidecar container in a script pod now observes pod status through one shared Kubernetes watch for the whole batch instead of one watch per script container, so its watch load and in-process state no longer grow with batch size — the suspected cause of sidecar OOMKills and of bursts of 410 Gone / too old resource version warnings on large batches. The sidecar also logs an observation summary while a batch runs and when it completes, reporting pod status events processed, watch reconnects, watch relists, peak goroutines, and, where the container's cgroup exposes it, peak memory in bytes, so a sidecar restart can be attributed without extra instrumentation. No configuration change is required; installs that raised the sidecar's memory limit can keep their current value. See Script runs survive an interrupted Kubernetes pod watch.

2026-08-25

Helm chart version: 3.18.0 Lunar images version: 3.18.0 Product updates included: 2026-08-25

Product updates

Migrations and upgrade notes

  • Drop any HUB_RETENTION_* entries from hub.extraEnv when you adopt hub.retention. extraEnv renders after the retention block, so keeping both emits the same variable names twice in the Hub container: a plain helm upgrade is last-wins and survives it — your extraEnv value still takes effect and the API server only warns that the earlier definition is hidden — but server-side apply rejects the object outright with duplicate entries for key [name="HUB_RETENTION_ENABLED"], so a GitOps install running ServerSideApply=true fails its next sync rather than degrading quietly. Installs that never set these through extraEnv are unaffected.

  • Database migrations now bound how long they wait for a table lock instead of queuing ahead of live traffic and blocking every new reader and writer of the table while they wait: the connection that applies them runs with lock_timeout = '5s', and the schema migrations that rework foreign keys on the Hub's largest tables — policy_runs, collection_records, and catalog_delta_contributions — allow 30 seconds. A contended migration therefore fails fast rather than stalling the upgrade or being killed as a deadlock victim minutes in: on a busy installation an attempt of the pre-rollout lunar-hub-migrate Job can end with canceling statement due to lock timeout (SQLSTATE 55P03) in its logs, the Job is retried according to its backoffLimit, and the upgrade proceeds once a lock is granted. Index builds are unaffected and still wait as long as they need.

  • Upgrading runs a database migration that normalizes stored pull-request states in place, folding GitLab's opened and locked values onto open; no operator action is required, and the migration writes nothing on a GitHub-only installation. On an installation connected to GitLab, expect a step change in scheduled collection volume — and in the script-runner capacity it consumes — on the first deploy, because cron collectors that declare runs_on: [prs] begin running against open merge requests. See Scheduled collectors run on open GitLab merge requests.

  • An installation that already sets a webhook_secret on an entry in HUB_GITHUB_APPS has existing repository hooks carrying the operator-level HUB_GITHUB_WEBHOOK_SECRET, which is no longer accepted for those owners after this upgrade: each affected repository's next delivery is rejected, Lunar re-stamps that repository's hook with the per-App secret, and its later deliveries succeed, so the fleet converges repository by repository as each one next sees activity. Redeliver anything lost in that gap from Settings → Webhooks → Recent Deliveries in GitHub, within GitHub's three-day window, and leave HUB_WEBHOOK_HEAL_MIN_INTERVAL at a non-zero value, since setting it to 0 turns the automatic repair off and the hooks then have to be deleted and recreated by hand. Installations that configure no per-App secrets need no action.

  • This upgrade runs database migrations that build the indexes retention needs on the commit, pull-request, repository, and configuration reference columns, and that re-declare two foreign keys so a delete in one retention tier cannot cascade into another. The indexes are built concurrently and the constraints are validated without an exclusive lock, so reads and writes continue throughout, but on a large database the migration job takes noticeably longer than usual to finish — allow for that before rolling the Hub.

  • This upgrade adds the indexes the retention sweep needs on the run-history tables: BRIN indexes on the timestamp column each table ages on, and btree indexes on four foreign-key columns pointing into them that were previously unindexed. They are built concurrently and never block reads or writes, but the migration job waits for the builds to finish, so on a large database this upgrade can take noticeably longer than a usual one.

Security

Per-App GitHub webhook secrets now apply to repository hooks

A webhook_secret set on an entry in HUB_GITHUB_APPS is now stamped on the repository webhooks Lunar creates for that App's owners, and repository deliveries from those owners are verified against it.

Previously the per-App value was consulted only for App-level deliveries, which Lunar never creates, so every repository hook carried the operator-level HUB_GITHUB_WEBHOOK_SECRET whichever App owned the repository. Two Apps configured with distinct secrets were therefore not isolated from one another: a secret leaked from one App validated deliveries forged for the other, which is the isolation the setting is documented to provide.

The secret is selected by the repository's owner and host, so the same owner name on github.com and on a GitHub Enterprise Server host do not share a secret. An owner whose App sets no per-App secret continues to use the operator-level HUB_GITHUB_WEBHOOK_SECRET, so an installation that configures no per-App secrets behaves exactly as before.

Features

Retention for superseded configuration and Git history

A second, opt-in retention tier removes what the runs window never reached: superseded Lunar configuration generations together with the components, domains, and script definitions published under them, plus commits, pull requests, and repositories. Set HUB_RETENTION_CASCADE_ENABLED=true alongside HUB_RETENTION_ENABLED — it defaults to false and stays off through an upgrade — and the Hub's periodic retention sweep prunes a configuration generation once it is both superseded and older than HUB_RETENTION_CONFIG, which defaults to 90d; 0 keeps every superseded generation. The newest generation for a repository is never pruned at any age, and no setting overrides that, so a repository always keeps the configuration its collectors and policies run against. A generation whose runs have not aged out yet is skipped and picked up on a later pass.

The same switch ages out commits on when they were committed and pull requests on when they were opened, against the HUB_RETENTION_RUNS window, and removes a repository only once nothing in the database still references it — its age acts only as a floor, so an active repository is never removed. Before a commit or repository is deleted, every table that names it is checked, including the many that name it by SHA or by identifier without a foreign key, so pruning cannot silently orphan data such as a component's current head commit.

This tier is kept behind its own switch because it deletes definitions rather than history, and it is the only part of retention that deletes a component.

Automatic repair after a GitHub webhook secret rotation

Rotating the GitHub webhook secret no longer leaves every already-installed hook signing with the old value. A hook keeps the secret it was created with, so after a rotation its deliveries fail signature validation and are rejected; the only remedy was deleting Lunar's hook on each repository so it would be recreated. The Hub now recognizes a delivery whose signature is well formed but does not verify, matches it to the hook it installed for that repository, and writes the currently configured secret onto that hook, so the next delivery succeeds.

Repairs run one at a time, each repository on its own next delivery, spaced by HUB_WEBHOOK_HEAL_MIN_INTERVAL — one second by default, so a fleet-wide rotation is not a burst of writes against GitHub's rate limits and a 30,000-repository install converges in roughly eight hours. Treat that rate as a floor: the Hub backs off whenever GitHub reports a rate limit, an allowance shared with the commit statuses and comments Lunar posts, so rotate a large fleet outside busy hours rather than shortening the interval. Setting the interval to 0 disables self-healing and leaves the manual delete-and-recreate procedure.

The delivery that reports the problem is still rejected, and GitHub does not retry it — replay it from the repository's Recent Deliveries within GitHub's three-day window. A repository with no activity stays on the old secret until something is pushed to it. GitLab is unchanged: its hooks carry no identifier that lets a failed delivery be attributed, so a rotated GitLab webhook secret still requires deleting each project's hook.

Data retention configured from chart values

Hub data retention is now configured through hub.retention chart values — enabled, window, configWindow, cascadeEnabled and vacuumEnabled, plus the pacing keys interval, batchSize, maxBatchesPerRun and vacuumLockTimeout — instead of raw HUB_RETENTION_* entries in hub.extraEnv.

Retention is off by default. An install that sets none of these keeps run history forever, exactly as before the block existed, so upgrading changes nothing on its own.

window sets HUB_RETENTION_RUNS and HUB_RETENTION_DERIVED to the same value from one key, deliberately: the Hub refuses to boot when the two differ, because derived surfaces rebuilt from the run tables would otherwise truncate themselves back to the runs window within a day.

Write windows in days. window and configWindow take Go durations extended with d (24h) and w (7d), so m is still Go's minutes — 12m is a twelve-minute window that passes every validation and deletes essentially all run history on the first sweep, and twelve months is 365d. The extension covers only those two keys: interval and vacuumLockTimeout are plain Go durations, so a daily sweep is 24h and interval: 1d renders happily but leaves the Hub crashlooping on unknown unit "d" after the migrate Job has already run.

cascadeEnabled and vacuumEnabled are separate switches because they delete more than run rows. vacuumEnabled is not gated by enabled: the compaction pass has its own queue and flag and never consults the master switch, so enabled: false with vacuumEnabled: true still compacts nightly. The three pacing keys are validated at boot even when enabled is false, and none may be zero.

Reclaiming deleted space with nightly compaction

Deleting run history caps database growth but does not shrink the files on disk — the freed space stays allocated to the table. The Hub can now hand it back to the operating system: set HUB_RETENTION_VACUUM_ENABLED=true and a nightly pass, scheduled by HUB_RETENTION_VACUUM_SCHEDULE (default 0 3 * * *), compacts at most one run-history table per night, worst first. Only the tables data retention deletes from are candidates; the queue tables are never compacted.

The schedule says when the pass looks, not when it acts. A table is compacted only when its estimated reclaimable space clears both HUB_RETENTION_VACUUM_MIN_BYTES (default 1 GiB) and HUB_RETENTION_VACUUM_MIN_RATIO (default 0.2, a fifth of the table), so on most nights it does nothing. Expect it to matter once, after retention first drains a large backlog.

Compaction holds an exclusive lock on the table it rewrites: queries against that table block for the duration and then complete, so choose a quiet window. HUB_RETENTION_VACUUM_LOCK_TIMEOUT (default 5s) bounds how long the pass waits for that lock before giving up until the next night, and HUB_RETENTION_VACUUM_TIMEOUT (default 2h) caps a single compaction. Neither accepts zero — the Hub refuses to start, because to Postgres a zero lock_timeout means waiting forever, which is how a bounded stall becomes an unbounded one.

Compaction is off by default, so an upgrade changes nothing until an operator turns it on.

Run-history retention

A Hub install can now bound the growth of its run history. Set HUB_RETENTION_ENABLED=true and the Hub periodically deletes script runs, policy runs, policy assertions and rollups, collection records, and catalog JSON items older than HUB_RETENTION_RUNS (default 90d). Retention is off by default and stays off through an upgrade, so nothing is removed until an operator opts in; on Kubernetes, set it through the hub.retention chart values described in Data retention configured from chart values.

Rows that something still references are kept whatever their age — a catalog JSON item still linked into the current catalog, for example — and are reconsidered on a later sweep once their dependants have aged out themselves.

Each sweep is rate limited, so turning retention on for the first time on an established install drains the accumulated backlog over days instead of in one pass: HUB_RETENTION_BATCH_SIZE (5000) and HUB_RETENTION_MAX_BATCHES_PER_RUN (100) bound the work of a single sweep, and HUB_RETENTION_INTERVAL (1h) sets how often it runs. The Hub logs what each sweep deleted, per table, along with budget_exhausted — while that is true there is still a backlog to work through, and when it turns false the install has caught up.

Deleting rows caps growth but does not return space to the operating system; the freed space becomes reusable by the same tables.

  • HUB_GITHUB_APPS and hub.github.apps now accept more than one App with the same owner. Give each entry its own App ID, installation ID, and private key; on Kubernetes, use privateKeyFile to select the matching key from appsSecret. See Multiple GitHub Apps for one organization and Avoid GitHub rate limiting.

  • New operator.scriptPodTopologySpreadConstraints chart value sets topologySpreadConstraints on every script pod the Operator creates, rendered as the JSON-encoded OPERATOR_SNIPPET_POD_TOPOLOGY_SPREAD_CONSTRAINTS environment variable, so script pods can be spread across zones, nodes, or subnets instead of piling onto whichever node the scheduler picks — the usual mitigation for lopsided per-subnet IP exhaustion. A constraint that omits labelSelector is filled in with the Operator's own app.kubernetes.io/managed-by: lunar-snippet-operator selector, because Kubernetes treats an absent selector as matching no pods, which would make the constraint a silent no-op; an explicit selector is left alone, so narrowing to one script type with lunar.earthly.dev/snippet-type still works. whenUnsatisfiable: ScheduleAnyway is the recommended setting: these pods are ephemeral and single-shot, and DoNotSchedule on a skewed cluster leaves them Pending until the pending-pod timeout and drops throughput. maxSkew, topologyKey, and whenUnsatisfiable are validated when the Operator starts — an invalid value fails startup, naming the offending constraint by index, rather than failing every pod creation later. Constraints are applied globally to all script pods; minDomains and whether the spread is actually satisfiable are not validated.

Improvements

  • hub.db.connectionOptions is now a map of Postgres connection parameters, and the chart renders the syntax each consumer expects, so the separator is no longer yours to get right; the default remains sslmode: require, and overriding replaces the whole map rather than merging into it. A new hub.db.sqlapiConnectionOptions sets the options the Hub hands to SQL API clients, surfaced by lunar sql — empty, the default, inherits hub.db.connectionOptions, so nothing about an existing install changes. Set it when SQL API clients reach Postgres by a different route than the Hub does, for example through a connection pooler on a hostname of its own that clients should verify with sslmode: verify-full and sslrootcert: system (which needs libpq 16 or newer; older clients need an explicit CA path). Keys render in sorted order rather than the order written, so migrating an existing multi-option string may reorder it, which nothing reads positionally.

  • An entry in hub.github.apps can now name the data key in appsSecret holding its PEM, with the optional privateKeyFile; it defaults to <lowercase-owner>.pem, so existing values render byte-for-byte unchanged. Set it to give one owner more than one GitHub App: GitHub's REST rate limit applies per App installation, so a second App on a busy org gives the Hub a second, independent budget, and two entries with the same owner would otherwise derive the same PEM filename and share a key. Requires Lunar Hub 3.14.0 or newer, which pools an owner's Apps and spreads read traffic across them while keeping commit statuses and pull-request comments on the owner's first entry; earlier Hubs reject a repeated owner at boot.

  • The Hub now reports the GitHub REST rate-limit budget of each GitHub App installation it uses, as the gauges lunar_github_rate_limit_remaining and lunar_github_rate_limit_limit on its /metrics scrape endpoint and over OTLP, so an exhausted budget is visible before the 403 responses it causes. Each sample is read from the rate-limit headers GitHub already returns on responses the Hub was making anyway — no extra API calls — and is labelled with the GitHub host, owner, App ID, and rate-limit resource only, deliberately without a per-repository label.

  • The Hub now prunes the Lunar configuration install directories it writes to its state volume after 10 days by default: HUB_INSTALL_FILE_MAX_AGE_DAYS now defaults to 10, matching the CLI's equivalent, instead of 0, which disabled age-based pruning entirely and left every generation on disk for the life of the install. Size-based pruning remains off, since HUB_INSTALL_FILE_MAX_DISK_SIZE still defaults to 0; set it if the volume is small enough that age alone could still fill it. To keep the previous behavior of never pruning by age, set HUB_INSTALL_FILE_MAX_AGE_DAYS to 0.

  • The HUB_RETENTION_RUNS window now also covers merged collection blobs — the merged component data stored per commit and component — aged on when the newest collection record folded into them arrived, so they no longer outlive the collection records they were built from.

  • The migrate Job may now retry 10 times before the release fails, up from a hardcoded 2, and the limit is settable as hub.migrateJobBackoffLimit. The retry exists for lock contention: changing a foreign key takes ACCESS EXCLUSIVE on both tables involved, so against a database still serving traffic a migration can lose a lock race, and the next attempt usually gets the lock. Ten is a ceiling rather than what an upgrade spends — Kubernetes backs off exponentially between attempts (10s, doubling, capped at 6m), landing them at roughly t = 0, 12, 32, 72, 152 and 312 seconds, so a default helm upgrade with its 5-minute timeout gets five attempts and raising --timeout buys the tail. A migration that is genuinely broken rather than unlucky still fails on the first attempt and every one after.

  • The lunar-hub-migrate pre-rollout Job now logs a line as it starts each migration, naming the migration file, its sequence, the direction, and the version table it belongs to, so an upgrade that is sitting inside one long-running statement can be told apart from one blocked on a lock. The queue-library migrations, which previously applied without any log output, are now logged on the same handler and level as the rest.

  • A database migration sets per-table autovacuum settings on the Hub's largest and most churn-heavy tables — script runs, policy runs, policy run rollups, the latest-policy-run table, policy assertions, collection records, and catalog JSON items — lowering their vacuum and analyze scale factors to 1% of the table with a 10,000-row floor. Until now these tables inherited the cluster defaults, whose proportional trigger meant the biggest tables were vacuumed least often and accumulated millions of dead tuples between passes; they are now vacuumed and analyzed more frequently and in smaller passes, which also keeps planner statistics closer to the data. The change is a catalog update that blocks neither reads nor writes and applies instantly regardless of table size; it runs automatically during the upgrade and requires no operator action. Installations that tune autovacuum for these tables at the cluster level should note that the per-table settings now take precedence.

  • Retention windows are now written in days: HUB_RETENTION_RUNS, HUB_RETENTION_DERIVED, HUB_RETENTION_CONFIG, and HUB_CATALOG_VERSION_RETENTION all default to 90d, and the Hub reports them back in the same units in logs and boot errors. Go durations such as 2160h still parse, so existing settings keep working, and a negative window is now rejected at startup instead of producing a cutoff in the future.

  • The Postgres coordinates the Hub hands out for the SQL API can now be overridden with the HUB_SQLAPI_HOST and HUB_SQLAPI_PORT environment variables, so lunar sql connection-string can point at an address the client actually reaches — a pooler or proxy in front of the database — rather than the host the Hub itself dials. Left unset, both fall back to the Hub's own database host and port, so an existing install vends the same connection string as before.

  • Session bounds can now be applied to the read-only sqlapi_user role: set HUB_SQLAPI_STATEMENT_TIMEOUT and HUB_SQLAPI_IDLE_IN_TRANSACTION_TIMEOUT (for example 60s and 120s) through hub.extraEnv, which the chart's hub-migrate Job inherits, since the migration is what applies them. Both are unset by default and unset means no bound, so an install that does not opt in keeps today's behavior; because the migration owns these role settings, clearing an environment variable removes the bound again, and a value set on the role by hand is cleared on the next migrate run. Setting a bound requires the Hub's database role to be able to ALTER ROLE — on a reduced-privilege install where sqlapi_user was pre-created and CREATEROLE was not granted, asking for a timeout fails the migrate Job rather than silently doing nothing. These bounds guard against accidental long-running or abandoned sessions only; a client can raise them again from its own session.

Bug fixes

  • hub.github.apps entries are now unique by host, owner and App ID instead of by owner alone, so the same owner name on github.com and on a GitHub Enterprise Server host is accepted — a combination the Hub has always supported and the chart previously rejected. Paired with that relaxation, two different Apps that resolve to the same PEM filename now fail template rendering with an error naming the file and pointing at privateKeyFile, instead of the second App signing its JWT with the first App's key. One App installed across several orgs still shares a key, which is legitimate.

  • A Hub pod now keeps serving against a database whose schema is ahead of its own binary, so a rolling upgrade no longer disrupts the not-yet-replaced pods. Because the pre-rollout lunar-hub-migrate Job completes before any new pod starts, a release that added a column left every old pod unable to read the affected tables — surfacing as errors such as failed to load current manifest: missing destination name … and taking down Lunar configuration loading, webhook ingestion, out-of-band collection, and scheduled cron collection for the length of the rollout, and indefinitely after a rollback to an older Hub image. The Hub's queries now name the columns they read instead of selecting every column, so an added column is ignored by an older binary.

  • A database migration that builds an index concurrently now drops any existing copy of that index first, so an upgrade whose migration Job is interrupted — by its deadline expiring or the pod being evicted mid-build — can simply be retried: the retry rebuilds the index instead of matching the invalid leftover, skipping the build, and recording the migration as applied over an index the planner will never use. An index already left invalid by an interrupted upgrade on an earlier version is not repaired by this change, because those migrations do not run again; such an index has to be dropped and re-created by hand.

  • The pre-install/pre-upgrade migrate Job now runs for up to 3600 seconds before Kubernetes kills it, up from a hardcoded 600, and the deadline is settable as hub.migrateJobActiveDeadlineSeconds. Ten minutes was sized as though it were a budget for how long migrations may take, so it fired on migrations that were healthy but slow — an index built with CREATE INDEX CONCURRENTLY scales with table size, and a 17 GB install spent roughly eight minutes inside a single concurrent build on a 2.1M-row table and failed the hook with DeadlineExceeded. The deadline is a backstop that reaps a wedged Job, needed because the migrator's advisory lock waits forever; how long an upgrade actually waits is your Helm timeout, so keep this value comfortably above it.

  • The nightly index rebuild on the Hub's and script Operator's internal job-queue tables now gets 15 minutes per index instead of the queue library's one-minute default, which a busy queue could not meet: REINDEX INDEX CONCURRENTLY waits out in-flight transactions before and after the build, so any transaction outliving the timeout aborted the rebuild. An aborted rebuild left an invalid <index>_ccnew index behind that Postgres kept maintaining on every job insert while the planner could not use it, and every later run then skipped that index permanently, so job-queue index bloat was never reclaimed. The timeout is configurable through HUB_QUEUE_REINDEXER_TIMEOUT and OPERATOR_REINDEXER_TIMEOUT. An installation that has already accumulated invalid _ccnew indexes needs a one-off cleanup after upgrading — confirm the original index each one shadows is still valid and ready, then DROP INDEX CONCURRENTLY the artifact — because the rebuild keeps skipping any index whose artifact is still present.

Deprecations

  • A plain string such as sslmode=require is still accepted for hub.db.connectionOptions and hub.db.sqlapiConnectionOptions and reaches every consumer verbatim, so an install upgrading from an earlier chart renders unchanged, but strings are removed in chart 4.0.0 — switch these values to maps. While you pass one, Helm logs warning: cannot overwrite table with non table, which is informational and does not stop the string being applied, and a string carrying more than one option is wrong for some consumers, since they do not all separate options the same way; the install notes flag it at upgrade time.

2026-08-19

Helm chart version: 3.13.2 Lunar images version: 3.13.2 Product updates included: 2026-08-19

Product updates

2026-08-18-2

Helm chart version: 3.13.1 Lunar images version: 3.13.1 Product updates included: 2026-08-18

Product updates

2026-08-18-1

Helm chart version: 3.13.0 Lunar images version: 3.13.0 Product updates included: 2026-08-18

Product updates

Migrations and upgrade notes

  • This release upgrades the Hub's internal job-queue library, and the pre-rollout lunar-hub-migrate Job applies its schema migration during the upgrade: it adds columns and indexes to the internal river_job table and backfills existing rows while holding locks on that table. The migration measured about 1.2 seconds per 400,000 job rows, so on an installation with a very large job backlog expect the migrate Job — which gates the rollout — to take correspondingly longer.

Improvements

  • Hub startup is much lighter on the database on large installations: the boot-time maintenance passes that keep derived check-run data consistent now resume from the last point they covered instead of re-scanning the entire run history on every boot of every replica — previously around fourteen minutes of full-table scanning per start on a large installation. The passes still run on every boot and still correct whatever they find; only the scan range is narrowed, and any interrupted or failed pass falls back to scanning more, never less.

  • A database migration drops five unused indexes from the table recording script runs (hub.snippet_runs), the most write-heavy table in the database, reducing its index storage — about 8 GB reclaimed on a large installation — and the write overhead of maintaining them. The migration runs automatically during the upgrade; no operator action is required.

2026-08-13

Helm chart version: 3.12.0 Lunar images version: 3.12.0 Product updates included: 2026-08-13

Product updates

Security

  • The Lunar images upgrade the golang.org/x/text dependency to v0.39.0, resolving the high-severity CVE-2026-56852 flagged by software-composition analysis; golang.org/x/sync moves to v0.21.0 as part of the same upgrade.

2026-08-12

Helm chart version: 3.11.0 Lunar images version: 3.11.0 Product updates included: 2026-08-12

Product updates

Features

  • The catalog versions backing the SQL API catalog view are pruned after a retention window set by the Hub's HUB_CATALOG_VERSION_RETENTION environment variable (default 2160h, 90 days; 0 disables pruning). The newest version is always kept regardless of age, so the view never goes empty. Each version stores a full catalog document — roughly 1 MB at 30,000 components — so the window bounds the history's disk usage. See Catalog history in the SQL API.

Improvements

  • Posting Lunar's check to a GitHub pull request no longer holds a database advisory lock — and the pooled database connection it pins — across GitHub API calls when a check run already exists for the commit; the lock is now taken only when the check run is first created. Under load on a large test installation, that lock accounted for over half of all database time, with connections parked on GitHub latency, so heavily loaded installations should see database connection pressure drop.

  • The Hub now logs a warning when a policy gate check arrives without a CI workflow run ID, naming the component, commit, and check type. Such a gate cannot exclude its own workflow run from the CI results it waits on and can block until its timeout when it shares a CI run with the component's collectors, so affected runs are now visible in the Hub logs instead of failing silently.

Bug fixes

  • The Hub no longer clones component repositories onto its own pod-local disk when dispatching code collectors or scheduled cron collector runs; the only checkout is the one made inside the script-runner pod, which was already the only one ever read. Previously these unused clones accumulated under /var/tmp/lunar/collectors on every Hub replica except the dispatching one — reaching tens of gigabytes on large installations, evicting Hub pods for exceeding ephemeral storage, and retaining repository source on Hub disks. One consequence for cron collectors: a branch that cannot be resolved now surfaces as a failed script run instead of being silently skipped with only a Hub log warning.

2026-08-11

Helm chart version: 3.10.0 Lunar images version: 3.10.0 Product updates included: 2026-08-11

Product updates

Improvements

  • Recoverable per-attempt failures during script log uploads are now logged as warnings, with only a terminal failure logged at error level, so transient upload retries no longer surface as error-level noise in log-based alerting.

2026-08-10

Helm chart version: 3.9.1 Lunar images version: 3.9.0 Product updates included: None

Features

Login-less Grafana viewing: grafana.anonymousViewer

The bundled Grafana can now be served with no login by setting grafana.anonymousViewer: true (default false). Unauthenticated visitors get the Viewer role in the default org and land straight on the dashboards — which is what makes the kiosk sidecar useful without handing out the admin password. The chart renders GF_AUTH_ANONYMOUS_ENABLED plus GF_AUTH_ANONYMOUS_ORG_ROLE=Viewer on the Grafana Deployment; the role is fixed by the template rather than read from values, so it cannot be widened to Editor or Admin by a typo, and the admin login form stays enabled for real Editor and Admin access.

This applies to grafana.mode: chart only: [auth.anonymous] is a Grafana server setting read at boot, so the chart can only apply it to the pod it owns. Setting it under external or off mode fails the install instead of silently doing nothing.

Only enable this when Grafana is not reachable from the internet. A Grafana Viewer can issue arbitrary queries against every provisioned datasource — restricting that is a Grafana Enterprise feature — so anonymous viewing grants anyone who can reach the Service read access to the whole Lunar database through the read-only datasource.

2026-08-07

Helm chart version: 3.9.0 Lunar images version: 3.9.0 Product updates included: 2026-08-07

Product updates

Features

First-class GitLab authentication

The Hub's GitLab credentials are now first-class chart values under hub.gitlab.*, replacing the documented hub.extraEnv + hub.volumes passthrough. Declare one hub.gitlab.tokens entry per top-level GitLab group ({group, host?, baseUrl?}), and put the group access tokens in a single operator-created Secret — named by hub.gitlab.tokensSecret.secretName, with one <lowercase-group>.token data key per entry — which the chart mounts at /secrets/gitlab, mirroring the multi-App GitHub pattern. A self-managed or GitLab Dedicated instance must be declared with host on its entry, or the Hub treats that host as GitHub.

The GitLab webhook signing secret is managed by hub.gitlab.webhookSecret: leave secretName empty and the chart generates a <release>-gitlab-webhook Secret that persists across upgrade and uninstall, or point it at a Secret you manage. Unlike GitHub, there is nothing to paste back anywhere — the Hub registers project hooks itself and stamps the secret on them. A per-entry tokens[].webhookSecret override remains available for advanced multi-instance isolation.

Improvements

  • The Dedicated setup documentation is restructured into three pages: an overview, a step-by-step setup guide that begins with private connectivity, and a standalone outbound PrivateLink guide written to be handed to a customer's networking team.

  • The chart no longer requires GitHub configuration: GitHub-only, GitLab-only, and mixed GitHub-and-GitLab values all render, as long as at least one SCM is configured, and a partial configuration for either SCM still fails at render time with a specific message. The chart-managed GitHub webhook secret and the GitHub entries in the install notes are rendered only when GitHub is configured; GitHub-only output is unchanged.

2026-08-03

Helm chart version: 3.6.1 Lunar images version: 3.6.0 Product updates included: None

Breaking changes

  • grafana.securityContext now applies only to the Grafana server container; the kiosk sidecar reads the new grafana.kiosk.securityContext value, which defaults to {}. If you set grafana.securityContext, copy the block to grafana.kiosk.securityContext when upgrading — otherwise the kiosk container silently loses those settings, and in a namespace with enforced Pod Security Standards the pod can be rejected outright.

Features

  • The bundled Grafana Deployment can now run more than one replica via grafana.replicaCount. Scaling beyond one requires pointing Grafana's backend store (sessions, organizations, annotations — distinct from the read-only dashboard datasource under grafana.provisioning.dbPassword) at Postgres through the new grafana.db values, which wire GF_DATABASE_* on the Grafana container; the chart fails at render time otherwise, since Grafana's default per-pod SQLite backend cannot be shared across replicas. The new grafana.topologySpreadConstraints value spreads replicas across nodes, defaulting to a soft node spread that is a no-op at a single replica.

Improvements

  • The kiosk sidecar in the Grafana pod — the nginx proxy that injects ?kiosk into dashboard URLs — is now configurable through grafana.kiosk: image.repository and image.tag (previously hardcoded to nginx:1-alpine, now defaulting to the pinned nginx:1.31.3-alpine so the build no longer drifts per node), resources, and securityContext.

  • All three containers in the Grafana pod (grafana, kiosk, and provision-reconverge) now have liveness and readiness probes, so Kubernetes detects and restarts a wedged container; previously none were probed.

  • The Grafana provisioning Job now honors grafana.annotations and grafana.podAnnotations, matching the existing Hub and Operator behavior.

2026-07-28

Helm chart version: 3.5.0 Lunar images version: 3.5.0 Product updates included: 2026-07-28

Product updates

Improvements

  • The Hub's http.app.server.duration metric no longer carries a per-repository github_repo attribute. On installations tracking many repositories, that attribute multiplied the metric's active series enough to overwhelm a Prometheus scraping the Hub; the bounded github_owner attribute remains.

  • Hub queue jobs that fail but still have retry attempts left now log at WARN; ERROR is reserved for a job's final, terminal attempt. An alert keyed on the Hub's queue job failed ERROR line now fires only when a job has exhausted its retries, not on transient failures that recover.

Bug fixes

  • The Hub no longer crash-loops at startup with bundle backfill failed ... configuration already exists after an earlier boot failed partway through installing the current Lunar configuration; the boot-time recovery now replaces the leftover on-disk copy instead of tripping over it on every subsequent start.

2026-07-21

Helm chart version: 3.4.2 Lunar images version: 3.4.2 Product updates included: 2026-07-21

Product updates

Improvements

  • The Operator's Kubernetes API client rate limits are now configurable through the OPERATOR_K8S_QPS and OPERATOR_K8S_BURST environment variables, with defaults raised to 50 QPS / 100 burst from the Kubernetes client's stock 5 / 10. The stock limit throttled script-run pod creation and cleanup, capping script execution throughput well below OPERATOR_MAX_CONCURRENT and node capacity; large installations can raise the new settings further to saturate more nodes.

2026-07-20-2

Helm chart version: 3.4.1 Lunar images version: 3.4.0 Product updates included: None

Features

  • New operator.scriptInitContainerSpec and operator.scriptSidecarContainerSpec Helm values set the container spec for the init and sidecar containers the Operator injects into each script pod (one of each, shared across all script types), with the same overlay rules as the operator.scriptContainerSpec* settings: you set resources, env, and so on, while the Operator overlays the image, env, and volume mounts. Set resources here so the init and sidecar containers satisfy a namespace ResourceQuota or LimitRange that requires requests; leaving them empty ({}) keeps the Operator's built-in defaults.

2026-07-20-1

Helm chart version: 3.4.0 Lunar images version: 3.4.0 Product updates included: 2026-07-20

Product updates

Bug fixes

  • A tracked repository that has no commits yet — created but never pushed — no longer causes repository sync to fail and retry indefinitely, flooding the Hub logs with errors on every reconcile. The Hub now treats an empty repository as a valid state on both GitHub and GitLab, logging a single warning and skipping it until commits arrive.

  • Script pods created by the Operator are no longer rejected in namespaces that enforce resource requests through a ResourceQuota or LimitRange (must specify requests.cpu for: init,sidecar): the init and sidecar containers the Operator injects into every script pod, which previously declared no resources, now carry small built-in CPU and memory requests and limits on every install — init 100m/128Mi requests with 500m/512Mi limits, sidecar 25m/64Mi with 100m/128Mi — independent of the script container's own, possibly much larger, resources. Clusters whose limits require different values can override them with the new OPERATOR_SNIPPET_INIT_CONTAINER_SPEC and OPERATOR_SNIPPET_SIDECAR_CONTAINER_SPEC environment variables on the Operator, each a JSON-encoded container spec of which only resources is applied; a malformed value fails the Operator at startup rather than at pod creation.

2026-07-17

Helm chart version: 3.3.0 Lunar images version: 3.3.0 Product updates included: 2026-07-17

Product updates

Improvements

  • The init and sidecar images of Operator-spawned script pods (lunar-snippet-init, lunar-snippet-sidecar) now run cleanly as a non-root user: each bakes a writable home directory (/home/lunar, owned by uid 1000 and group-writable by the root group), so a non-root runAsUser set through operator.scriptPodSecurityContext or operator.scriptContainerSpec* — as required by a cluster's restricted Pod Security Standards — no longer fails on writes under $HOME, and no custom-built images are needed. The images still start as root by default; nothing changes unless a non-root security context is configured.

Bug fixes

  • A container-level securityContext configured through the operator.scriptContainerSpec* chart values now applies to every container in a script pod the Operator creates — the init and sidecar containers as well as the script containers. Previously only the script containers carried it, so clusters enforcing the "restricted" Pod Security Standard flagged the init and sidecar containers for not dropping capabilities, which a pod-level securityContext cannot fix because fields like capabilities exist only at the container level. No configuration change is needed, and installs that set no container-level securityContext are unaffected.

2026-07-14

Helm chart version: 3.2.1 Lunar images version: 3.2.0 Product updates included: None

Features

  • Operator pods can now be spread across nodes or zones with operator.topologySpreadConstraints — set it when running more than one Operator replica. It is empty by default, so no constraints are applied unless configured.

Improvements

  • The Hub's pre-install/pre-upgrade database migration Job now honors hub.annotations and hub.podAnnotations, applying them to the Job and its pod to match the annotation controls already available on the long-running components.

2026-07-13

Helm chart version: 3.2.0 Lunar images version: 3.2.0 Product updates included: 2026-07-13

Product updates

Security

  • The dashboards deploy image, ghcr.io/earthly/lunar-dashboards, now builds its bundled grpcurl from source with a current Go toolchain instead of shipping the upstream prebuilt binary, clearing the fixable critical CVEs image scanners flagged in it (Go standard-library CVEs including CVE-2024-24790 and CVE-2025-22871, and GHSA-p77j-4mvh-x3m3 in google.golang.org/grpc). Upgrading the chart pulls the patched image; the image's behavior is unchanged.

Features

  • New operator.scriptPodAnnotations and operator.scriptPodSecurityContext chart values set pod annotations and the pod-level securityContext (fsGroup, runAsUser, seccompProfile, …) on every script pod the Operator creates, rendered as the JSON-encoded OPERATOR_SNIPPET_POD_ANNOTATIONS and OPERATOR_SNIPPET_POD_SECURITY_CONTEXT environment variables. The pod-level securityContext is the piece typically required to run script pods under a cluster's "restricted" Pod Security Standards; container-level security context remains configurable separately via operator.scriptContainerSpec*. A malformed JSON value fails the Operator at startup instead of at pod creation.

Bug fixes

  • Grafana dashboard provisioning — the ghcr.io/earthly/lunar-dashboards image the chart runs as a deploy Job and as the Grafana pod's reconverge sidecar — no longer fails at pod cold start when the Hub has not yet provisioned the Grafana database connection or Grafana's dashboard.grafana.app/v2 API server is still registering. It now waits for both, bounded by HUB_RESOLVE_TIMEOUT and V2_READY_TIMEOUT (120 seconds each by default), instead of failing with dashboards reconverge failed until the next pod restart, and it fails fast with a clear error when the target Grafana is older than 12.0, the minimum version for the v2 dashboard schema.

  • On installations running more than one Hub replica, SQL API materialization no longer runs concurrently on every replica — previously each replica rebuilt the same tables at the same time, adding redundant database load. At most one materialization now runs across the whole installation, and any additional refresh, including a manual lunar sql refresh, queues behind the one in progress.

2026-07-11

Helm chart version: 3.0.0 Lunar images version: 3.0.0 Product updates included: 2026-07-11

Product updates

Migrations and upgrade notes

Grafana now runs the stock upstream server, with dashboards deployed over its API

Lunar no longer ships a custom Grafana server image. The chart now runs the stock grafana/grafana server, and Lunar's panel plugins, datasources, and dashboards are installed into it over Grafana's HTTP API by a new deploy image, ghcr.io/earthly/lunar-dashboards, which the chart runs as a deploy Job on install and upgrade. The same image can target a Grafana you operate yourself — including Grafana Cloud — resolving the Grafana endpoint and database connection from the Hub.

Database migrations now additionally create grafana_user, a read-only Postgres role that backs the Grafana datasource. It is created during the normal migration run and is covered by the same CREATEROLE privilege already required for sqlapi_user. The datasource connects with sslmode=require by default; plain Postgres without TLS must set HUB_GRAFANA_DB_CONNECTION_OPTIONS to sslmode=disable.

Two operational notes: Grafana plugins are now installed at pod start from grafana.com instead of being baked into the server image, so restricted-network installs need egress to grafana.com or plugins pre-installed out of band; and ghcr.io/earthly/lunar-dashboards is a private image, pulled with the same imagePullSecrets as the other Lunar images.

  • Hub database migrations in this release re-key the materialized_checks table to one row per applying policy import and rename historical duplicate-named policy imports to stable names (see Stable names for duplicate policy imports), and clear the checks materialization watermark, so the first materialization run after upgrading rebuilds checks data from the full history; on large installations that run can take noticeably longer than usual. Run that rebuild with a single Hub replica — temporarily set hub.replicaCount: 1 and scale back once the pass completes — so that replicas do not rebuild the same data concurrently. See One check row per applying policy import.

2026-07-09

Helm chart version: 2.17.0 Lunar images version: 2.8.0 Product updates included: 2026-07-09

Product updates

Security

  • Patches golang.org/x/net and OpenSSL CVEs across all published images (Hub, Grafana, Operator, init, sidecar): yq is bumped to 4.53.3 and the Alpine base to 3.23.5. Upgrading the chart pulls the patched images.

2026-07-08-2

Helm chart version: 2.16.0 Lunar images version: 2.7.0 Product updates included: None

Features

  • The Operator Role now grants coordination.k8s.io/leases (the events grant already existed) — the leader-election RBAC required when operator.replicaCount > 1 on a leader-election-capable Operator image. The Lease lives in the Operator's script namespace, so no new Role or namespace is needed.

  • New operator.replicaCount (default 1) runs the Operator at two or more replicas. On an Operator image with Manager leader election (2.8.0 or later), script execution stays active-active across replicas while exactly one replica runs the pod-GC reconciler via a leader-election Lease; on an earlier image, more than one replica is still safe but runs duplicate (idempotent) GC passes. Chart 2.16.0 defaults to Operator 2.7.0, before leader election; override operator.image.tag with 2.8.0 or upgrade to chart 2.17.0 before raising operator.replicaCount, and account for Operator Postgres connections scaling with the replica count.

2026-07-08-1

Helm chart version: 2.15.0 Lunar images version: 2.7.0 Product updates included: 2026-07-07

Product updates

2026-07-06

Helm chart version: 2.14.0 Lunar images version: 2.6.0 Product updates included: None

Migrations and upgrade notes

  • Upgrading without pinning hub.replicaCount moves an existing install from one Hub replica to two, doubling its pod and Postgres connection footprint. Each replica budgets about 85 connections (the hub.db.maxOpenConns / maxPoolConns / operatorPoolSize defaults of 40/40/5), so size Postgres max_connections to at least replicaCount × 85 plus headroom for the migrate Job, psql, and superuser-reserved slots — about 200 for the two-replica default. Managed Postgres (RDS, Aurora, Cloud SQL) typically provisions well above this, but a bare Postgres at the default max_connections=100 must be raised before upgrading. Do not front the Hub with a transaction-mode connection pooler — it breaks the Hub's session advisory locks and LISTEN/NOTIFY.

Improvements

  • The Hub now runs highly available by default: hub.replicaCount changes from 1 to 2 and hub.podDisruptionBudget.enabled from false to true (maxUnavailable: 1), so a default install tolerates losing one Hub pod and keeps a replica serving during voluntary node drains. Scale to 3 for more headroom, or set hub.replicaCount: 1 (and disable the PodDisruptionBudget) for a minimal single-instance install — switching between the two is config-only, with no schema or state change either way.

2026-07-03

Helm chart version: 2.13.0 Lunar images version: 2.6.0 Product updates included: None

Improvements

  • The Hub pod now runs a preStop hook (hub.preStopSleepSeconds, default 10 seconds) that sleeps before SIGTERM, giving Kubernetes time to deregister the pod from the Service so new requests stop arriving before the Hub drains; set it to 0 to disable. hub.terminationGracePeriodSeconds (default 60) covers this sleep plus the Hub's shutdown budget (HUB_SHUTDOWN_TIMEOUT, 45s).

  • The Hub readiness probe now targets /ready (previously /health) with failureThreshold: 1, so the pod reports not-ready as soon as the Hub begins graceful shutdown and Kubernetes deregisters it. Liveness still targets /health, which is process-only, so a draining or briefly database-blocked pod is never restarted. Requires a Hub image that serves /ready — 2.6.0 or later, the chart's new default — so keep the chart and image versions in lockstep if you pin image tags.

2026-07-02-3

Helm chart version: 2.12.0 Lunar images version: 2.5.0 Product updates included: None

Breaking changes

Hub storage becomes ephemeral; the hub-data PVC is removed

The Hub's /var/lib/lunar directory is now an ephemeral emptyDir instead of a PersistentVolumeClaim. The Hub keeps no durable state on disk — only re-extracted runtimes and a rebuildable script-code cache, with script code served from S3 — so the ReadWriteOnce hub-data PVC is removed, along with the hub.persistence values (enabled, storageClass, size, accessModes). A new hub.stateDir.sizeLimit value (default 2Gi) caps the ephemeral volume. With no shared volume to contend for, the Hub Deployment now rolls out with RollingUpdate at any replica count — the Recreate strategy and the 2.11.0 render-time guard that forced persistence off for multi-replica deployments are both gone.

Upgrading does not delete the existing <release>-hub-data PVC (it was created with helm.sh/resource-policy: keep); it is simply orphaned, so remove it manually to reclaim storage. Before upgrading, make sure the Hub is already running an image that serves script code from S3 (hub 2.5.0 or later, the chart default since chart 2.5.0) and has pulled the Lunar configuration at least once on such a version, so script bundles exist in S3 when new pods start with cold, empty state directories.

  • The hub.rootDir value and the HUB_ROOT_DIR environment variable it rendered are removed. The Hub has no such configuration field, so the value never had any effect; delete it from your values file if present.

2026-07-02-2

Helm chart version: 2.11.0 Lunar images version: 2.5.0 Product updates included: None

Features

Multi-replica Hub baseline

The chart can now run the Hub with more than one replica. New values: hub.replicaCount (default 1, unchanged behavior), hub.terminationGracePeriodSeconds (default 60), hub.topologySpreadConstraints (default none), and an optional hub.podDisruptionBudget (default off, because a PodDisruptionBudget on a single replica can block voluntary node drains — enable it for HA).

A soft node spread applies by default (maxSkew: 1 across kubernetes.io/hostname with ScheduleAnyway), so multi-replica deployments survive a node loss without extra configuration; override it via hub.topologySpreadConstraints. Running more than one replica requires hub.persistence.enabled=false, since Hub state lives in Postgres: the chart fails fast at render time when replicaCount > 1 with persistence still enabled, instead of leaving replicas contending for a single ReadWriteOnce PersistentVolumeClaim. Rendering also fails if the PodDisruptionBudget sets both minAvailable and maxUnavailable, a combination the Kubernetes API rejects.

2026-07-02-1

Helm chart version: 2.10.0 Lunar images version: 2.5.0 Product updates included: 2026-07-02

Product updates

Features

  • New hub.annotations, grafana.annotations, and operator.annotations values render onto each workload's Deployment metadata, distinct from the existing podAnnotations values that apply to the pod template. hub.annotations and grafana.annotations previously existed as chart values but were never wired into a template; operator.annotations is new. All default to empty, so existing installs are unchanged.

2026-07-01-5

Helm chart version: 2.9.0 Lunar images version: 2.5.0 Product updates included: None

Features

  • New hub.db.maxOpenConns, hub.db.maxPoolConns, and hub.db.operatorPoolSize values render to HUB_DB_MAX_OPEN_CONNS, HUB_DB_MAX_POOL_CONNS, and HUB_MAX_OPERATOR_POOL_SIZE. The defaults match the Hub's existing built-in values (40, 40, and 5), so single-replica installs are unchanged. Total Postgres connection demand is replicaCount × (maxOpenConns + maxPoolConns + operatorPoolSize): multi-replica deployments should lower maxOpenConns and maxPoolConns to fit the server's max_connections, keep maxPoolConns at or above peak concurrent workers (the sum of hub.maxWorkers.*) to avoid serializing store queries, and consider a connection pooler at larger replica counts.

2026-07-01-4

Helm chart version: 2.8.0 Lunar images version: 2.5.0 Product updates included: None

Features

  • New hub.grpc.maxConnectionAge and hub.grpc.maxConnectionAgeGrace values (defaults 30m and 10m) render to HUB_GRPC_MAX_CONNECTION_AGE and HUB_GRPC_MAX_CONNECTION_AGE_GRACE. Bounding connection age makes long-lived HTTP/2 clients reconnect periodically, so they redistribute across Hub replicas as the fleet scales and drop off a draining replica during a rollout instead of staying pinned to the replica they first reached. Clients transparently re-resolve on GOAWAY, so the defaults are safe to keep; set both to 0 to disable connection cycling on single-instance installs. Requires a hub image with these settings (2.5.0 or later, the chart default) — older images ignore the environment variables.

2026-07-01-3

Helm chart version: 2.7.0 Lunar images version: 2.5.0 Product updates included: None

Bug fixes

  • Fresh installs no longer fail on the hub-migrate Job. As a pre-install hook, the Job runs before the chart's ServiceAccount exists, so referencing that ServiceAccount left the Job unschedulable until it timed out and helm install failed with DeadlineExceeded. The migrator only talks to Postgres, so it now runs under the namespace's default ServiceAccount. Upgrades were never affected — the ServiceAccount already existed from the prior release — and image pulls are unaffected because imagePullSecrets is set on the pod spec.

2026-07-01-2

Helm chart version: 2.6.0 Lunar images version: 2.5.0 Product updates included: None

Bug fixes

  • The hub-migrate Job now renders hub.extraEnv, matching the Hub Deployment, so environment variables such as HUB_SQLAPI_PASSWORD — the only way the chart supplies SQL API credentials — also reach the migrator. Previously, on a fresh database the migration created the sqlapi_user role without a password, leaving the SQL API unable to authenticate. Only first-time installs were affected; an already-migrated database does not re-run the migration.

2026-07-01-1

Helm chart version: 2.5.0 Lunar images version: 2.5.0 Product updates included: None

Migrations and upgrade notes

Database migrations run as a pre-upgrade Job

Database migrations now run as a Helm pre-install/pre-upgrade hook Job, not at Hub boot. The hub-migrate Job runs /bin/lunar-hub-migrate once per release — before the Hub Deployment is updated — so migrations complete first and gate the rollout. The Hub server now asserts at boot that the database schema is current and refuses to start if it is behind, which also makes scaled-up or restarted pods safe: they never run migrations themselves. This replaces the previous boot-time, per-pod in-process migration and is a prerequisite for running the Hub with more than one replica.

The migrate Job requires Hub 2.5.0, the first release that ships /bin/lunar-hub-migrate. The chart and Hub versions must move together: with an older pinned Hub image the Job cannot run because the binary is absent, and Hub images from 2.5.0 onward no longer migrate at boot, so they require this chart version or an equivalent external migration step.

2026-06-24

Helm chart version: 2.4.1 Lunar images version: 2.4.1 Product updates included: 2026-06-23

Product updates

Security

  • The 2.4.1 images carry high-severity Go dependency security updates (golang.org/x/crypto, golang.org/x/net, golang.org/x/sys, jackc/pgx). Upgrading the chart pulls the patched images.

Features

  • Buildkite ingestion is disabled by default for Self-hosted installations. To enable it, set HUB_BUILDKITE_WEBHOOK_TOKEN, expose a reachable Hub webhook endpoint, configure Buildkite to send events there, and use a CI/CD Tracer version with Buildkite support. See Buildkite support.

Improvements

  • Operator logs now identify the script and component behind a script-runner pod OOM kill and report the memory limit it exceeded, so operators can identify which workload needs more memory.

2026-06-17-2

Helm chart version: 2.4.0 Lunar images version: 2.3.1 Product updates included: None

Features

  • New optional hub.github.apps[].host and hub.github.apps[].baseUrl render a GitHub Enterprise Server host and API endpoint into the matching HUB_GITHUB_APPS entry. Both default to github.com behavior and are omitted when empty, so existing github.com-only multi-App configurations render unchanged. Chart 2.4.0 still defaults to Hub 2.3.1, which accepts the values but does not provide end-to-end GHES App authentication; use Hub 2.4.1 or upgrade to chart 2.4.1 for GHES token minting and per-host App keying.

2026-06-17-1

Helm chart version: 2.3.1 Lunar images version: 2.3.1 Product updates included: 2026-06-17

Product updates

Improvements

  • The static, image-coupled GF_* settings (GF_INSTALL_PLUGINS, GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS, GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH, GF_USERS_ALLOW_SIGN_UP, GF_USERS_DEFAULT_THEME, GF_FEATURE_TOGGLES_ENABLE) moved out of the Grafana Deployment env and into the lunar-grafana image, so they version with the image and no longer shadow it. Override per-deployment via grafana.extraEnv. Requires a lunar-grafana image with these settings baked in — this chart release's default image (2.3.1) has them.

2026-06-05

Helm chart version: 2.3.0 Lunar images version: 2.2.1 Product updates included: None

Features

  • New operator.scriptPodPriorityClassName (default "") sets the priorityClassName on every script pod the Operator creates, rendered as OPERATOR_SNIPPET_POD_PRIORITY_CLASS_NAME; empty leaves the field unset, so pods take the cluster's default priority. The referenced PriorityClass must already exist in the cluster. Useful for ensuring script pods are terminated before service pods in single-namespace setups, or for prioritizing workloads in shared scratch namespaces.

2026-05-27

Helm chart version: 2.2.1 Lunar images version: 2.2.1 Product updates included: 2026-05-27

Product updates

Improvements

  • A new partial B-tree index on snippet_runs (started_at DESC) speeds up narrow-window Runs-dashboard queries (roughly 300× on a started_at-only filter, 6.5× on a one-day panel render). The Hub's migration runner applies it on startup; fresh environments build the index inline with a brief ShareLock on snippet_runs.

Bug fixes

  • Three Runs dashboard fixes in Grafana: the [collectors]/[policies] links from the component dashboard now land on populated rows, policy script names in the Queued tab render as clickable links, and the Created column displays relative time. Ships in lunar-grafana 2.2.1, this chart release's default Grafana image.

2026-05-26-2

Helm chart version: 2.2.0 Lunar images version: 2.1.1 Product updates included: None

Features

  • Multi-App GitHub authentication: the new hub.github.apps list pairs each {owner, appId, installId} entry with a per-owner PEM key in a Secret you manage, referenced by hub.github.appsSecret; the chart renders the HUB_GITHUB_APPS JSON and mounts the Secret at /secrets/github-apps. Runtime support requires Hub 2.2.0 or newer, but chart 2.2.0 still defaults to Hub 2.1.1; override hub.image.tag or upgrade to chart 2.2.1. The legacy single-App hub.github.app.* configuration remains supported, and render-time validation enforces mutual exclusivity between the two modes.

2026-05-26-1

Helm chart version: 2.1.0 Lunar images version: 2.1.1 Product updates included: None

Features

  • New hub.secrets.<scope>.perKey (default false) switches script-secret delivery for a scope from a single HUB_<SCOPE>_SECRETS environment variable to per-key injection that surfaces each Secret data key as HUB_<SCOPE>_SECRET_<KEY>, so one key can be rotated or added with kubectl patch secret without re-supplying the others. The Hub merges both shapes when both are configured (per-key wins on conflict), so keys can be migrated one at a time. This requires Hub 2.2.0 or newer, but chart 2.1.0 still defaults to Hub 2.1.1; override hub.image.tag or upgrade to chart 2.2.1 before enabling it.

2026-05-20

Helm chart version: 2.0.0 Lunar images version: 2.1.1 Product updates included: None

Breaking changes

  • Webhooks can now be exposed on a separate ingress from the API: hub.ingress is reshaped into hub.ingress.api and hub.ingress.webhooks, and the chart computes per-component URLs from the right ingress. The previous single-host ingress configuration is no longer accepted — upgrading from 1.x fails at render time with migration messages: move host under both api.host and webhooks.host, and grpcAnnotations/httpAnnotations under api.grpcAnnotations/api.httpAnnotations (see the README section "Migrating from chart 1.x").

  • hub.grafanaURLBase is renamed to grafana.externalURL, and rendering fails with a migration message if the old key is still set. The effective Grafana URL — rendered as HUB_GRAFANA_URL_BASE for the Hub's dashboard links and as Grafana's own GF_SERVER_ROOT_URL — now resolves from grafana.externalURL, then a chart-managed Grafana ingress host, and is otherwise left empty rather than guessed from the public webhook host, fixing OIDC redirect-URI breakage on split-host installs. Installs that expose Grafana through external routing must set grafana.externalURL explicitly.

  • hub.publicBaseURL is replaced by hub.webhookURL, which is now optional: with chart-managed ingress it defaults to https://<hub.ingress.webhooks.host>, while bring-your-own-ingress installs must set it explicitly. Rendering fails with a migration message if the old key is still set.

2026-05-18

Helm chart version: 1.0.2 Lunar images version: 2.1.1 Product updates included: None

Features

  • New hub.maxWorkers.{collect,policy,cronCollect,cataloger} values surface the Hub's HUB_MAX_WORKERS_* worker-concurrency caps through the chart. Defaults are 10/20/5/1 respectively; 0 means unlimited. This requires Hub 2.2.0 or newer, but chart 1.0.2 defaults to Hub 2.1.1, which ignores these variables; override hub.image.tag or upgrade to chart 2.2.1 to apply the caps.

2026-05-17-2

Helm chart version: 1.0.1 Lunar images version: 2.1.1 Product updates included: None

Bug fixes

  • The broken v2.1.1 image default is corrected to 2.1.1; chart 1.0.0's defaults were not pullable without an override because the v-prefixed tag did not exist in the registry.

2026-05-17-1

Helm chart version: 1.0.0 Lunar images version: v2.1.1 Product updates included: 2025-10-20, 2025-12-19, 2026-05-15, 2026-05-17

Product updates

Breaking changes

  • Default image pulls switched from Docker Hub (earthly/lunar-*) to GitHub Container Registry (ghcr.io/earthly/lunar-*). Installations that pin Docker-Hub-only dev-build SHAs should point the image repository values back at Docker Hub.

  • Lunar Hub no longer supports the legacy HUB_GITHUB_TOKEN personal-access-token authentication path. Before upgrading, create and install a GitHub App and configure its owner, App ID, installation ID, and PEM private key through hub.github.app.*; the Hub refuses to start without complete App credentials.

  • Chart 1.0.0 replaces floating main image tags with a pinned default, but its exact v2.1.1 tag was not published to GitHub Container Registry. This release therefore requires an image-tag override; chart 1.0.1 corrects the default to 2.1.1.

  • Chart values and templates now use "script" terminology in place of "snippet": the operator.snippet* keys (such as snippetNamespace, snippetContainerSpec*, snippetPodNodeSelector, and snippetPodTolerations) are gone — set the new script* equivalents instead. "snippet" survives only in the image names themselves.

Features

Lunar Helm chart reaches 1.0

Chart 1.0.0 concludes the early-access 0.x line, which ran for 26 releases from 0.1.0 (2026-02-09) through 0.8.2 (2026-05-17). Over those releases the chart grew from a Hub-plus-Grafana deployment into the full self-hosted stack: in-cluster execution of collectors and policies through the Operator, node-scheduling and image-pull controls, Hub persistence and public-URL configuration, chart-managed secrets and a required tenant ID, Grafana enabled by default with Postgres TLS support, and licence configuration.

The 0.x charts used floating main image tags from Docker Hub and had no production users according to the chart changelog. Chart 1.0.0 begins the release-by-release history and switches to versioned image defaults.

Licence-based deployment

Self-hosted Lunar installations are provisioned with a signed licence: a JWT issued by Earthly that carries the tenant identity, telemetry configuration, and optionally a container-registry pull credential. Lunar Hub verifies the licence at startup, applies the configuration it carries in place of hand-configured tenant and telemetry settings, and fails closed when the licence is missing, invalid, or expired.

PreviousSelf-hosted
NextCLI and CI/CD Tracer

Last updated