Data Retention
What a Lunar Hub install deletes on its own, what it keeps for the life of the install, and the environment variables that change either.
What a Lunar Hub install deletes on its own, what it keeps forever, and which knobs change that.
Do you need this page?
Read it if you operate a Hub and want to bound its storage growth, or if you need to state a retention window to satisfy a compliance requirement. If you run Lunar Dedicated, Earthly sets these for your install and you can skip to What no window covers to see what retention does not reach.
This page is about data the Hub stores. It is not about the CI Tracer's local files, which the CLI configuration reference covers.
The short version
Run history is capped by a retention window, but the feature is off until you turn it on. Set HUB_RETENTION_ENABLED=true and the Hub ages out runs, policy results, and collection records older than 90 days. Leave it off, the default, and those tables grow with your commit volume and nothing removes them.
Queue rows, catalog history, and on-disk working files are capped separately and always have been.
Run and trend retention
The customer-facing contract is two windows rather than a knob per table, because there are only two questions to answer: how far back can you see a run, and how far back do your trends go.
HUB_RETENTION_ENABLED
false
Master switch. Nothing below deletes anything while this is off
HUB_RETENTION_RUNS
90d
Script runs, policy runs, policy assertions and rollups, collection records, catalog JSON items, merged collection blobs
HUB_RETENTION_DERIVED
90d
Materialized trends, component scores, the runs listing
HUB_RETENTION_CONFIG
90d
Superseded manifest generations, and the components, domains and script definitions they defined
HUB_RETENTION_CASCADE_ENABLED
false
Whether to prune those generations at all
Windows are written in days — 90d, 365d. Go durations such as 2160h still parse, but days are what these knobs mean and the Hub reports them back that way.
The runs and derived windows must be equal in this version, and the Hub refuses to boot if they are not. Setting a longer derived window looks like it should work and does not: two of the derived surfaces are rebuilt from the run tables, so they truncate themselves back to the runs window within a day, silently. Rejecting the config at boot is better than accepting one the Hub cannot deliver. Divergence becomes supported once those surfaces are fixed.
Pruning superseded config
Every time a repository's config is published, the Hub writes a new manifest generation with its own components, domains, and script definitions. Those accumulate, and the components underneath them are the bulk of it.
HUB_RETENTION_CASCADE_ENABLED turns on removing them, and it is separate from HUB_RETENTION_ENABLED because it is a different kind of deletion. The runs window removes history — rows describing something that happened. This removes the definitions those rows point at, and it is the only part of retention that deletes a component.
A generation is pruned when it is both superseded and older than HUB_RETENTION_CONFIG. It has its own window rather than following the runs window, because config churn and run history are kept for different reasons.
The newest generation for a repository is never pruned, at any age. It is that repository's live configuration, so removing it would leave collectors and policies with nothing to run against. There is no setting that overrides this.
A generation whose runs have not aged out yet is skipped rather than pruned, and picked up on a later pass once the runs sweep has caught up.
Retention is off by default and stays off through an upgrade. Turning it on for the first time on an established install has years of accumulated rows to remove, so it deliberately does not do that in one pass. HUB_RETENTION_BATCH_SIZE (5000) and HUB_RETENTION_MAX_BATCHES_PER_RUN (100) bound each run, and HUB_RETENTION_INTERVAL (1h) sets how often it runs. The backlog drains over days. Watch the Hub logs for budget_exhausted — while it is true there is still a backlog, and when it goes false the install has reached steady state.
Commits, pull requests, and repositories
Aged on their own clocks rather than the runs window, because they are not runs. Commits age on when they were committed, pull requests on when they were opened, and repositories on when Lunar last synced them — a repository is an identity, so "not seen in the window" is the condition that means it is gone, where "created long ago" only means it is old. An active repository is never removed for age.
A commit is removed only once nothing still names it. That check is wider than it looks: thirteen tables reference a commit by its SHA and only two by a foreign key, so the database alone would not stop a delete that orphans, for example, a component's current head pointer. All of them are checked before a commit is removed.
Reclaiming deleted space
Deleting rows caps growth. It does not shrink the database file: the freed space becomes reusable by that table, and stays allocated.
Handing it back to the operating system takes a compaction pass, which holds an exclusive lock on the table while it runs. Readers block for the duration and then continue — it is a stall, not an error — but on a large table that stall is measured in minutes, so it is off by default and wants a maintenance window.
HUB_RETENTION_VACUUM_ENABLED
false
Master switch for compaction
HUB_RETENTION_VACUUM_SCHEDULE
0 3 * * *
When it checks, not when it acts
HUB_RETENTION_VACUUM_MIN_BYTES
1073741824 (1 GiB)
Minimum reclaimable space to justify the stall
HUB_RETENTION_VACUUM_MIN_RATIO
0.2
Minimum fraction of the table that is reclaimable
HUB_RETENTION_VACUUM_LOCK_TIMEOUT
5s
How long to wait for the lock before giving up until tomorrow
HUB_RETENTION_VACUUM_TIMEOUT
2h
Cap on one compaction
It runs nightly and should do nothing on almost every run. A table is compacted only when it clears both thresholds, and only one table is compacted per night. Expect it to matter once, after retention first drains a large backlog, and rarely after that.
Leave the lock timeout short. The exclusive lock queues behind whatever transaction is already running, and every query that arrives meanwhile queues behind it, so a compaction that waits patiently is how a stall becomes an outage. Giving up and retrying tomorrow costs nothing.
Queue retention
Terminal jobs are the highest-churn rows in the system. The Hub's queue and the script operator's queue each keep their own, and each is tuned separately.
HUB_QUEUE_COMPLETED_JOB_RETENTION
24h
Jobs that finished successfully
HUB_QUEUE_CANCELLED_JOB_RETENTION
24h
Jobs cancelled before completing
HUB_QUEUE_DISCARDED_JOB_RETENTION
168h
Jobs that exhausted their retries
OPERATOR_COMPLETED_JOB_RETENTION
24h
As above, for the script-execution queue
OPERATOR_CANCELLED_JOB_RETENTION
24h
Script-execution jobs cancelled before completing
OPERATOR_DISCARDED_JOB_RETENTION
168h
Script-execution jobs that exhausted their retries
Nothing reads a terminal job. The durable record of a run is hub.snippet_runs, so shortening these windows loses no history you can query. Discarded jobs are the exception worth keeping longer, since they are failures someone may want to inspect.
Reach for the operator knobs first on a large fleet. operator_queue.river_job takes one row per script-execution batch and churns harder than any other table.
Catalog history
HUB_CATALOG_VERSION_RETENTION (default 90d) bounds how long captured catalog versions are kept for the SQL API catalog view. Set it to 0 to disable pruning.
Each version stores a complete catalog document, roughly 1 MB at 30,000 components, so the window multiplies against your capture rate rather than your component count alone. The newest version is always kept regardless of age, so the view never goes empty.
On-disk working files
These govern the Hub's volume, not the database.
HUB_INSTALL_FILE_MAX_AGE_DAYS
10
Manifest directories, pruned by age
HUB_INSTALL_FILE_MAX_DISK_SIZE
0 (off)
The same directories, pruned by total size
HUB_BUNDLE_MAX_AGE
2h
Policy bundles staged for execution
HUB_BUNDLE_NO_DELETE
false
Keeps bundles for debugging
The Hub writes a directory per manifest generation, so on a busy config this is the volume's fastest-growing tenant. The size arm is off by default because a byte cap only makes sense against a known volume size. Set it if your volume is small enough that age alone could still fill it.
HUB_BUNDLE_MAX_AGE covers a scratch directory rather than the persistent volume, so it is rarely the knob you want.
Script pods
Pods are reaped by the operator on a separate set of timers: OPERATOR_TERMINAL_POD_GRACE_PERIOD (5m), OPERATOR_MAX_POD_AGE (12h), OPERATOR_RUNNING_POD_TIMEOUT (6h), and OPERATOR_PENDING_POD_TIMEOUT (10m). OPERATOR_RETAIN_FAILED_PODS keeps failed pods for inspection and is off by default.
Object storage
Lunar never deletes from your buckets. Run logs and run-bundle archives stay until something outside Lunar removes them. Nothing on this page changes that: the windows above govern database rows, not objects.
On a self-hosted install, set S3 lifecycle rules yourself, and check them against your own retention obligations first, since both buckets can contain credentials, script source, or PII surfaced from CI runs. The self-hosted prerequisites cover the bucket layout. On Lunar Dedicated the buckets live in your own account, so the rules are still yours to set — agree them with Earthly during setup rather than assuming either side has.
What no window covers
HUB_RETENTION_RUNS reaches the run tables and their dependants. These are outside it, at any setting:
The newest manifest generation for each repository, and everything under it
Everything else is covered, though not all of it by the runs window: components, domains, and script definitions go with the generation that defined them under HUB_RETENTION_CONFIG, and commits, pull requests, and repositories have their own clocks. Both need HUB_RETENTION_CASCADE_ENABLED.
What 0 means
0 is not one sentinel. It means different things to different knobs on this page, and several of them mean the opposite of what you would expect.
Knob
0 means
HUB_RETENTION_RUNS / _DERIVED
Keep forever. Retention is disabled
HUB_RETENTION_CONFIG
Keep every superseded generation
HUB_INSTALL_FILE_MAX_AGE_DAYS
Keep forever. The pruner returns early
HUB_CATALOG_VERSION_RETENTION
Keep forever. Pruning is disabled
HUB_QUEUE_* / OPERATOR_* job retention
River's own default — 24h, 24h, 168h. Not forever
HUB_BUNDLE_MAX_AGE
Delete every bundle on the next pass, including one staged for execution
Pod timeouts (OPERATOR_*)
Reap immediately
HUB_RETENTION_VACUUM_LOCK_TIMEOUT / _TIMEOUT
Rejected at startup. Not a sentinel — to Postgres a zero lock_timeout means wait forever, which is the behaviour the compaction section argues against
The queue knobs are the trap worth naming, because the advice above is to keep discarded jobs longer than the rest: setting HUB_QUEUE_DISCARDED_JOB_RETENTION=0 to keep them indefinitely silently gets you seven days. To keep terminal jobs forever, set -1ns. River's infinite sentinel is -1, and the value is parsed as a duration, so a bare -1 is rejected for having no unit.
To keep bundles, use HUB_BUNDLE_NO_DELETE=true rather than a zero age.
Sizing
Two things drive Hub database growth, and only one of them scales with the size of your fleet.
Run history scales with commit volume, bounded by the retention window. A component that never changes costs almost nothing; one receiving fifty commits a day writes runs fifty times a day. Estimate from your commit and merge rate over one window, not from your component count. With retention off, drop the window and the figure has no ceiling.
Catalog history scales with capture rate and component count together, at roughly 1 MB per captured version at 30,000 components.
The queue tables are bounded by the retention windows above and reach a steady state. If they do not, that is a backlog rather than a retention problem, and the queue-depth metrics on the Hub's /metrics endpoint will show it.
Related
Day 2 Operations — upgrades, secret rotation, backup, and the diagnostics bundle
Prerequisites — database and bucket sizing before install
Last updated